Tag Archives: tips

MIT Course Majors Male Female Ratio, Fall 2011 (scraping with google docs)

as seen at stata; click to see larger pic

when i saw this in stata a few days ago i legitimately wondered for a few seconds which way this was color-coded until I saw a note (cropped out in this pic), which made me smile. At MIT and hopefully in the general world the imperfect gender ratios are really not so noticeable unless you’re looking for it. MIT is wonderful (I say this now because I have no finals this term, all project classes that I loved even if I didn’t do as well as I’d’ve liked)

Anyway, there were no citations! so I decided to go off and investigate on my own and treat it as a data visualization exercise (hi, upcoming IAP / startlabs / c2c / metrify).

EDIT: Derp! I data-failed. You can see civil/environmental engineering (course 1 — see http://web.mit.edu/catalog/subjects.html if you’re confused by the numbers) is definitely off. : / messed up copying and pasting somewhere. Will redo this for realz one day (they have data for multiple years, I want to clean it up automatically).

sorted by %
sorted by course number


data source
Here’s the data I used (I should just email them for the csv’s, but I thought this would be a simple data scraping exercise):
What tripped me up is that the women vs. everyone listing of majors is ever so slightly off, and I have to decide how many distinctions I want to make. Do I really care about 2 vs 2A? not really. If I care about 6-1 vs 6-2 vs 6-3, do I just drop 6-7?

assorted google spreadsheets of data from those sources
cleaned data for charts

raw data sorted for female / undergards

raw scraped html table data

where i tried to (see script editor) automagically clean up the data and failed:

google apps code 
I used for that last gdoc

function loadImport() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getActiveSheet();
var data_sskey = "0Atr0HZeoMbmgdFYzNUdEN2NKM2lGbHV6OVN4YkJNTmc";
var row_init = "13";
var col_ug = "G";
var col_grad = "M";
var col_total = "O";
var col_courseName = "A";
var col_courseNumber = "Q";

var cols = [col_ug, col_grad, col_total];
//create arrayformulas for copying selected data over
//e.g. =ImportRange("0Atr0HZeoMbmgdFYzNUdEN2NKM2lGbHV6OVN4YkJNTmc","Women!G13:G")

//Add Course Names
var finalRanges = [ '=ImportRange("' + data_sskey + '","Women!' + col_courseName + "13:" + col_courseName+ '")'];

//Add Enrollment Data in alternating female / all
for (var i = 0; i < cols.length; i++){
femRange = '=ImportRange("' + data_sskey + '","Women!' + cols[i] + "13:" + cols[i]+ '")';
everyoneRange = '=ImportRange("' + data_sskey + '","Everyone!' + cols[i] + "13:" + cols[i]+ '")';

//Add Course Numbers
finalRanges.push('=ImportRange("' + data_sskey + '","Women!' + col_courseNumber + "13:" + col_courseNumber+ '")');

// Sets the column header names to match copied data.
var columnNames = ['Course Name', 'Fem UG', 'All UG', 'Fem Grad', 'All Grad', 'Fem Total', 'All Total', 'Course Number'];
var headersRange = sheet.getRange(1,1,1,columnNames.length);
//var dataRange = sheet.getRange(2,1,1,columnNames.length);
for (var j = 0; j < columnNames.length; j++){
sheet.getRange(2, j+1).setValue(finalRanges[j]);

//method getRange(row, column, optNumRows, optNumColumns)

Google Docs Scripting Tutorials
An excellent step-by-step free tutorial on ImportXML (similar to importHTML that I used), how I heard of xpath (which I didn’t end up need to use due to importHTML magic): http://www.distilled.net/blog/distilled/guide-to-google-docs-importxml/
other useful tutorials I referred to
http://blog.fastfedora.com/2011/08/how-to-use-open-apis.html *shudders* speaking of government apis, I really should be less terrible and fix my github codeforamerica repos which I left half-finished. AUGH. I FAIL. This is why I haven’t applied to any jobs yet. Upcoming: post on why you shouldn’t hire me (with the caveat that I feel okay listing these because these are past mistakes that I don’t intend to make again).
eeee shiny http://pipes.yahoo.com/pipes/
I also attempted to use google fusion tables but did not have a good experience.
To investigate: http://www.propublica.org/nerds/item/using-google-refine-for-data-cleaning

eagle persistent hotkeys / emergency h bridge WIP

to get
persistent keyboard shortcuts / hotkeys in eagle, 
since eagle’s UI leaves something to desired for me
you need to add lines to the eagle.scr file in the “scr” folder of your eagle installation
e.g. for my windows desktop that is: “C:Program Files (x86)EAGLE-5.11.0screagle.scr
here’s what mine’s set at:

# Configuration Script
# This file can be used to configure the editor windows.

# eagle Configuration Script — persistent hotkeys
# nancy ouyang
# 13 dec 2011
Assign F1 ‘wire’;
Assign F2 ‘move’;
Assign F3 ‘del’;
Assign F4 ‘group’;
Assign F5 ‘name’;
Assign F6 ‘label’;
Assign F7 ‘value’;
Assign F8 ‘copy’;

Assign AF1 ‘route’;
Assign AF2 ‘ratsnest’;
Assign AF3 ‘rip’;

emergency high-power h bridge
is in progress and due in oh 18 hours or so (still need to finish schematicking). because i’m actually rich (i live off of instant noodles because i’m a cheap student and would much rather spend the money on delicious components for projects), or at least i feel rich when immersed in the general uh resourceful atmosphere of MITERS / east campus, i bought delicious power components. (as opposed to relying on 6.131 lab ones).

anyway so I emergency learned myself some theory. as usual with my schoolwork, the path of this project went “i’m really lost, but too proud to ask for help so let me confuse myself on the internet”. several days later, i feel distinctly antisocial and hopeless and take lots of naps. a few days before the deadline, i start trying to do something and gradually i go “whoa it’s starting to make sense.” then i go “whoa maybe there is hope after all! I should try to complete this in time” followed by “AAAHHH this is due in [too little time] emergency project!!1!1!!”

current WIP progress (clearly not finished and there are errors, need more buscap, mislabeled components,  etc) —

i referenced shane colton’s hexbridge / other motor controllers and charles guan’s tinytroller / other motor controllers.

h bridge links
that i used to read up on everything i was supposed to already know:

this is a good speedy intro to why+ theory + practicals of making your own h-bridge (n+p mosfet design, as opposed to nmosfet-only w/ driver)

more links here

===misc links===
using chip http://www.robotroom.com/HBridge.html + thermal data http://www.robotroom.com/Thermal-PCB-Motor-Driver-Tests-4.html
steppers http://www.massmind.org/techref/io/stepper/hipwrbp-gm.htm
dc motors overview http://www.electronics-tutorials.ws/io/io_7.html
robot power http://www.robotpower.com/downloads/
stepper drivers http://www.nerdkits.com/forum/thread/694/
more reference yay open source hardware http://www.robotpower.com/products/osmc_info.html
stepper and motor control math!theory http://www.divms.uiowa.edu/~jones/step/circuits.html
lots of different h bridge implementations (schematics) http://circuitelec.blogspot.com/2009/07/h-bridge-motor-driver-circuit.html
look, half bridge can be reversible if you have negative rails (terminology ftw) http://www.me.berkeley.edu/ME102/lab3.html
steppers i have access to http://store.makerbot.com/nema-17-stepper-motor.html
because the awesome built in delay fet driver http://www.irf.com/product-info/datasheets/data/ir2184.pdf
==more specific to me links==
Half-Bridge Circuit Behavior by Dennis L Feuch

wish me luck…

Hardware Search Engines

Previously I daydreamed about octopart for meche; turns out I spoke too hastily. There are supplies search engines which exist already, though they’re not as cool as octopart IMHO ๐Ÿ™‚ Well, none of them have customer reviews like google shopping anyway.

While hunting for spring suppliers (as opposed to making the springs myself, which may work but I’m not sure), I ran across globalspec and then I used one of those “similar sites” search engines to find several hardware supplier search engines.

In rough order of relevance / how much I liked them:

  • http://www.thomasnet.com/prodsearch.html
    • Search ThomasNet to find manufacturers, distributors and service providers – from Actuators to Zirconium and everything in between.
    • geographical, not just by state but by “within x miles”
    • has whitepaper and some explanations at taxonomic levels — on the K-12 level sort of explanation
  • http://www.globalspec.com/
    • The Engineering Search Engine
    • has a taxonomy of sorts “browse products and suppliers”
    • get “register” nagging :///
    • has geographical locations with maps, so I can search for local suppliers of springs
    • as an “application notes” search, presented as search results (not nicely), and also 1-sentence explanations of things (although has descriptions at a slightly “deeper” level than thomasnet, I feel)
  • http://www.engnetglobal.com/
    • EngNetยฎ … to enable Engineers, Technicians, Tradesmen, etc. to find information and communicate effectively with suppliers in the Engineering Industry. […] provides a searching facility free of charge to users
    • more varied taxonomies / ways to browse (by industries, by engineering discipline, products/services, brandnames)
    • no “extra explanation” details, solely listing
  • http://www.amazon.com/industrial-scientific-supplies/b?ie=UTF8&node=16310091
    • pointed out to me by octopart person brian newbold yay
      • http://www.directindustry.com/ 
        • The Virtual Industrial Exhibition — more 
        • also has taxonomy
        • more of an index of companies
        • no geographical
        • http://www.engineering.com/Products.aspx
          • directory of companies
        • http://www.engineersedge.com/engineering/Products_Directory/
          • ugly interface, no specific search, and too many ads, IMHO
        • http://www.interfacebus.com
          • web 1.0 Text link directly ftw. Taxonomy + search via google. Cleaner than engineers edge (less ads all over the place)
        Engineering reference sites:
        • http://www.efunda.com/home.cfm Efunda
          • eFunda stands for engineering Fundamentals. Its mission is to create an online destination for the engineering community, where working professionals can quickly find concise and reliable information to meet the majority of their daily reference needs.
          • Nicely designed interface and information is cleanly presented ๐Ÿ˜€ makes me want to nom nom info
        • http://www.engineeringtoolbox.com/
        • http://kmoddl.library.cornell.edu/index.php
          • Kinematic models! Oh man, I can look at pretty mechanisms online. <3 delicious.
        • http://www.electronics-tutorials.ws/
          • because the awesome. so true, what that “thank-you” video guy said re: Khan’s videos and what advantages videos offer over humans (besides freeing up time for knowledge-bearers to not just lecture but help people one on one):
            • And some of those videos I watched over and over twenty of thirty times until I got it. I don’t think there’s a tutor in the world that I could pay who would repeat the same thing twenty or thirty times without getting at least a little bit judgmental.
          • asdlfkj yes. This, professors, is why — even if it encourages people to be truant — why lectures should be recorded. Guess what? I’m not going to get it the first time. 

        Also, octopart competition? o__o  Sort of.

          • http://www.findchips.com/
            • “Welcome to the original and fastest electronic part finder and chip search engine. FindChips electronic components search engine scours the leading distributors to give you instant inventory levels and pricing around the world.”
            • Well designed, displays key data very cleanly and good for checking component specs, not just sourcing suppliers, but clearly for people who know exactly what they want. Does not provide octopart’s (perhaps amazon inspired?) extremely flexible and awesome search-by-specification that allows you to browse components
          random link dumps:
          http://engineeryourlife.org/ designed to convince girls to be engineers
          spring suppliers, search:
          http://www.hardwareproducts.com/ — MA but rather bad online catalog browsing interface
          http://www.planetspring.com — online spring design calculator, not quite sure how it links to ordering yet, seems to have a weekly or monthly fee? *confused*
          and linking this again b/c I keep forgetting to read it: RFID explained http://www.morganclaypool.com/doi/abs/10.2200/S00040ED1V01Y200602MPC001
          also, a bit late for me, but some electromagnet force calculators
          still need to learn how to use FEMM