News for hackers. Stuff that matters.
Re:Protip for salary seekers: H1-B filings are public by tzs on Wednesday April 13, 2011 @11:08PM (#2445407)
For you Jedi out there, the headache you are feeling right now is thousands of copies of Excel 2008 on Macs crashing as people try to make pivot tables from this data.

My 2009 Mac Pro with 8 GB of RAM was defeated by the 2010 file.

Re:Protip for salary seekers: H1-B filings are public by chussad on Wednesday April 13, 2011 @11:24PM (#2445440)
in case this helps, this is for the year 2010 data create table visa_status (

LCA_CASE_NUMBER text, STATUS text, LCA_CASE_SUBMIT text, VISA_CLASS text, LCA_CASE_EMPLOYMENT_START_DATE text, LCA_CASE_EMPLOYMENT_END_DATE text, LCA_CASE_EMPLOYER_NAME text,

LCA_CASE_EMPLOYER_ADDRESS text,

LCA_CASE_EMPLOYER_CITY text,

LCA_CASE_EMPLOYER_STATE text,

LCA_CASE_EMPLOYER_POSTAL_CODE text,

LCA_CASE_SOC_CODE text,

LCA_CASE_SOC_NAME text,

LCA_CASE_JOB_TITLE text,

LCA_CASE_WAGE_RATE_FROM text,

LCA_CASE_WAGE_RATE_TO text,

LCA_CASE_WAGE_RATE_UNIT text,

FULL_TIME_POS text,

TOTAL_WORKERS text,

LCA_CASE_WORKLOC1_CITY text,

LCA_CASE_WORKLOC1_STATE text,

PW_1 text,

PW_UNIT_1 text,

PW_SOURCE_1 text,

OTHER_WAGE_SOURCE_1 text,

YR_SOURCE_PUB_1 text,

LCA_CASE_WORKLOC2_CITY text,

LCA_CASE_WORKLOC2_STATE text,

PW_2 text,

PW_UNIT_2 text,

PW_SOURCE_2 text,

OTHER_WAGE_SOURCE_2 text,

YR_SOURCE_PUB_2 text,

LCA_CASE_NAICS_CODE text,

Decision_Date text);

LOAD DATA LOCAL INFILE '/H1B_2010_TEXT/H1B_FY2010.txt' INTO TABLE visa_status FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n';

Re:Protip for salary seekers: H1-B filings are public by nutjob123 on Thursday April 14, 2011 @12:01AM (#2445536)
I haven't opened the file but this sounds strange. I've reached the excel cell number limit before crashing computers with much less than 8gb of ram. http://office.microsoft.com/en-us/excel-help/excel-specifica...
Re:Protip for salary seekers: H1-B filings are public by smokinn on Thursday April 14, 2011 @12:52AM (#2445674)
Creating a pivot table is very different from simply opening an excel file. Conceptually (it may be more optimized internally) it tries to create a cube of all possible results with all others. (An OLAP cube.)

So if you have columns 1, 2, 3, 4 and rows 1, 2, 3 it will create a cube for a combination of all dimensions.. So for all rows you will have all columns instantly queryable but also 1-2, 1-3, 1-4, 2-3, 2-4, 3-4, 1-2-3, 1-2-4, 1-3-4, 2-3,4, 1-2-3-4 so that you can query any combination of rows/cells and instantly get your result.

As you can imagine this cube gets exponentially bigger as you add dimensions (columns) and will quickly kill excel if you load in too much stuff.