MIT Course Majors Male Female Ratio, Fall 2011 (scraping with google docs)

as seen at stata; click to see larger pic

when i saw this in stata a few days ago i legitimately wondered for a few seconds which way this was color-coded until I saw a note (cropped out in this pic), which made me smile. At MIT and hopefully in the general world the imperfect gender ratios are really not so noticeable unless you’re looking for it. MIT is wonderful (I say this now because I have no finals this term, all project classes that I loved even if I didn’t do as well as I’d’ve liked)

Anyway, there were no citations! so I decided to go off and investigate on my own and treat it as a data visualization exercise (hi, upcoming IAP / startlabs / c2c / metrify).

EDIT: Derp! I data-failed. You can see civil/environmental engineering (course 1 — see if you’re confused by the numbers) is definitely off. : / messed up copying and pasting somewhere. Will redo this for realz one day (they have data for multiple years, I want to clean it up automatically).

sorted by %
sorted by course number

data source
Here’s the data I used (I should just email them for the csv’s, but I thought this would be a simple data scraping exercise):
What tripped me up is that the women vs. everyone listing of majors is ever so slightly off, and I have to decide how many distinctions I want to make. Do I really care about 2 vs 2A? not really. If I care about 6-1 vs 6-2 vs 6-3, do I just drop 6-7?

assorted google spreadsheets of data from those sources
cleaned data for charts

raw data sorted for female / undergards

raw scraped html table data

where i tried to (see script editor) automagically clean up the data and failed:

google apps code 
I used for that last gdoc

function loadImport() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getActiveSheet();
var data_sskey = "0Atr0HZeoMbmgdFYzNUdEN2NKM2lGbHV6OVN4YkJNTmc";
var row_init = "13";
var col_ug = "G";
var col_grad = "M";
var col_total = "O";
var col_courseName = "A";
var col_courseNumber = "Q";

var cols = [col_ug, col_grad, col_total];
//create arrayformulas for copying selected data over
//e.g. =ImportRange("0Atr0HZeoMbmgdFYzNUdEN2NKM2lGbHV6OVN4YkJNTmc","Women!G13:G")

//Add Course Names
var finalRanges = [ '=ImportRange("' + data_sskey + '","Women!' + col_courseName + "13:" + col_courseName+ '")'];

//Add Enrollment Data in alternating female / all
for (var i = 0; i < cols.length; i++){
femRange = '=ImportRange("' + data_sskey + '","Women!' + cols[i] + "13:" + cols[i]+ '")';
everyoneRange = '=ImportRange("' + data_sskey + '","Everyone!' + cols[i] + "13:" + cols[i]+ '")';

//Add Course Numbers
finalRanges.push('=ImportRange("' + data_sskey + '","Women!' + col_courseNumber + "13:" + col_courseNumber+ '")');

// Sets the column header names to match copied data.
var columnNames = ['Course Name', 'Fem UG', 'All UG', 'Fem Grad', 'All Grad', 'Fem Total', 'All Total', 'Course Number'];
var headersRange = sheet.getRange(1,1,1,columnNames.length);
//var dataRange = sheet.getRange(2,1,1,columnNames.length);
for (var j = 0; j < columnNames.length; j++){
sheet.getRange(2, j+1).setValue(finalRanges[j]);

//method getRange(row, column, optNumRows, optNumColumns)

Google Docs Scripting Tutorials
An excellent step-by-step free tutorial on ImportXML (similar to importHTML that I used), how I heard of xpath (which I didn’t end up need to use due to importHTML magic):
other useful tutorials I referred to *shudders* speaking of government apis, I really should be less terrible and fix my github codeforamerica repos which I left half-finished. AUGH. I FAIL. This is why I haven’t applied to any jobs yet. Upcoming: post on why you shouldn’t hire me (with the caveat that I feel okay listing these because these are past mistakes that I don’t intend to make again).
eeee shiny
I also attempted to use google fusion tables but did not have a good experience.
To investigate:

eagle persistent hotkeys / emergency h bridge WIP

to get
persistent keyboard shortcuts / hotkeys in eagle, 
since eagle’s UI leaves something to desired for me
you need to add lines to the eagle.scr file in the “scr” folder of your eagle installation
e.g. for my windows desktop that is: “C:Program Files (x86)EAGLE-5.11.0screagle.scr
here’s what mine’s set at:

# Configuration Script
# This file can be used to configure the editor windows.

# eagle Configuration Script — persistent hotkeys
# nancy ouyang
# 13 dec 2011
Assign F1 ‘wire’;
Assign F2 ‘move’;
Assign F3 ‘del’;
Assign F4 ‘group’;
Assign F5 ‘name’;
Assign F6 ‘label’;
Assign F7 ‘value’;
Assign F8 ‘copy’;

Assign AF1 ‘route’;
Assign AF2 ‘ratsnest’;
Assign AF3 ‘rip’;

emergency high-power h bridge
is in progress and due in oh 18 hours or so (still need to finish schematicking). because i’m actually rich (i live off of instant noodles because i’m a cheap student and would much rather spend the money on delicious components for projects), or at least i feel rich when immersed in the general uh resourceful atmosphere of MITERS / east campus, i bought delicious power components. (as opposed to relying on 6.131 lab ones).

anyway so I emergency learned myself some theory. as usual with my schoolwork, the path of this project went “i’m really lost, but too proud to ask for help so let me confuse myself on the internet”. several days later, i feel distinctly antisocial and hopeless and take lots of naps. a few days before the deadline, i start trying to do something and gradually i go “whoa it’s starting to make sense.” then i go “whoa maybe there is hope after all! I should try to complete this in time” followed by “AAAHHH this is due in [too little time] emergency project!!1!1!!”

current WIP progress (clearly not finished and there are errors, need more buscap, mislabeled components,  etc) —

i referenced shane colton’s hexbridge / other motor controllers and charles guan’s tinytroller / other motor controllers.

h bridge links
that i used to read up on everything i was supposed to already know:

this is a good speedy intro to why+ theory + practicals of making your own h-bridge (n+p mosfet design, as opposed to nmosfet-only w/ driver)

more links here

===misc links===
using chip + thermal data
dc motors overview
robot power
stepper drivers
more reference yay open source hardware
stepper and motor control math!theory
lots of different h bridge implementations (schematics)
look, half bridge can be reversible if you have negative rails (terminology ftw)
steppers i have access to
because the awesome built in delay fet driver
==more specific to me links==
Half-Bridge Circuit Behavior by Dennis L Feuch

wish me luck…

CNC Petri BioArt, introduction (no build yet)

it occurs to me that i should explain what bioart is, for those who’ve never heard of it

or specifically, what i mean in terms of my project: “bioart” = streaking petri dishes with bacteria the glow (fluoresce or bioluminesce).

people have done it by hand for ages (this site used for reference for DIY broth too)

and these are the different colors i wanted to use (clearly, it’s end of term so i won’t have time this semester…)

and to use a pen-plotter system (xy axis gantries) + appropriate stepper software to streak petri dishes by computer instead of by hand.

look ma, i’m all lasercutter!

one of the original inspirations… hunter cole, her living dna:

bright enough to light a room / picture (not sure about exposure they used though)

and of course, there’s already CNC bioart (with a 3d printer frameset), on agar with drops of flourescent bacteria

and recently one with lots of different colors:

some on paper instead of agar

it’s everywhere… here are some misc. links

Digikey has product guides that I’d never seen before (thanks mas.863 classmates)
~~6.131 motor controller notes
notes to self. ir2125 wants 230 mV min, attiny can source max 40 mA per pin

spring making: worked on spring-making some more for vending machine.

AVR Studio Screenshots / overview of features (doesn’t go over the chip debugger, which I haven’t learned to use it)