Re:Show HN: Explore 16 Years of Green Card Applications
by negrit on Wednesday June 03, 2015 @10:44PM
(#9657154)
PostgreSQL :)
Re:Show HN: Explore 16 Years of Green Card Applications
by bootload on Thursday June 04, 2015 @09:56PM
(#9663431)
so just dump the data in and process via sql. What about pre-processing ie: different data sources (pdf, cvs), incomplete or overlapping data? I imagine some code had to be written to do this?
Re:Show HN: Explore 16 Years of Green Card Applications
by negrit on Friday June 05, 2015 @04:28PM
(#9667910)
All my datasource where csv and mdb files. So, I directly imported them to PostgreSQL. Very little code was written to clean the data. 99% of the cleaning was done with SQL queries.
The code is mostly used to display the data. Cleaning the data with SQL queries is much faster than writing code.
What tools did you use/craft to clean the source data?