News for hackers. Stuff that matters.
Re:Show HN: Explore 16 Years of Green Card Applications by negrit on Wednesday June 03, 2015 @10:00AM (#9652549)
Hello everyone. I quickly built this little tool based on public records provided by the government. I did the same a couple weeks ago with H-1Bs and I just added over 1.1million green cards records.

The data provided is not perfect so i'm still working on cleaning it but it should give you an idea of what is going on.

If you have any questions, feel free to ask.

Re:Show HN: Explore 16 Years of Green Card Applications by the_economist on Wednesday June 03, 2015 @12:04PM (#9653537)
This is awesome Theo. When can I have data about Americans in France?
Re:Show HN: Explore 16 Years of Green Card Applications by Jake232 on Wednesday June 03, 2015 @10:15AM (#9652669)
I can't seem to find the H1B statistics page, maybe I'm just looking in the wrong place though. Got a link?
Re:Show HN: Explore 16 Years of Green Card Applications by bootload on Wednesday June 03, 2015 @09:56PM (#9657003)
"If you have any questions, feel free to ask."

What tools did you use/craft to clean the source data?

Re:Show HN: Explore 16 Years of Green Card Applications by negrit on Wednesday June 03, 2015 @10:44PM (#9657154)
PostgreSQL :)
Re:Show HN: Explore 16 Years of Green Card Applications by bootload on Thursday June 04, 2015 @09:56PM (#9663431)
so just dump the data in and process via sql. What about pre-processing ie: different data sources (pdf, cvs), incomplete or overlapping data? I imagine some code had to be written to do this?
Re:Show HN: Explore 16 Years of Green Card Applications by negrit on Friday June 05, 2015 @04:28PM (#9667910)
All my datasource where csv and mdb files. So, I directly imported them to PostgreSQL. Very little code was written to clean the data. 99% of the cleaning was done with SQL queries.

The code is mostly used to display the data. Cleaning the data with SQL queries is much faster than writing code.

Re:Show HN: Explore 16 Years of Green Card Applications by SuperKlaus on Wednesday June 03, 2015 @01:28PM (#9654319)
Great tool! There seems to be a bug though when I drill down to year plus country of citizenship and then try to filter by job description and state or city. The job description search term ends up in the state input field.

Also, filter by company would be great!

Re:Show HN: Explore 16 Years of Green Card Applications by negrit on Wednesday June 03, 2015 @01:53PM (#9654517)
My bad.

Fixing it ASAP

Re:Show HN: Explore 16 Years of Green Card Applications by anorark on Wednesday June 03, 2015 @07:14PM (#9656497)
This looks great! Can the green card data be shown by categories? Like how many green cards applications were filed in each category like EB1, EB2, etc.
Re:Show HN: Explore 16 Years of Green Card Applications by jlees on Wednesday June 03, 2015 @01:28PM (#9654321)
Neat tool, though it's a little disconcerting to be able to easily find my application - not your fault, though, the data was already there!
Re:Show HN: Explore 16 Years of Green Card Applications by dalek2point3 on Wednesday June 03, 2015 @02:28PM (#9654749)
kinda weird that all the timeseries charts have time going backwards. going lower to higher is certainly the standard way to do it. for eg: http://data.jobsintech.io/companies/google-inc
Re:Show HN: Explore 16 Years of Green Card Applications by netheril96 on Wednesday June 03, 2015 @10:38AM (#9652860)
Any way for us to run `select` queries ourselves?
Re:Show HN: Explore 16 Years of Green Card Applications by negrit on Wednesday June 03, 2015 @10:52AM (#9652978)
I haven't planned it but why not. I don't really know how to do it though.
Re:Show HN: Explore 16 Years of Green Card Applications by roddit on Wednesday June 03, 2015 @10:15AM (#9652670)
Kudos my friend.
Re:Show HN: Explore 16 Years of Green Card Applications by Oras on Wednesday June 03, 2015 @10:35AM (#9652838)
Great work mate