Re:US Census API now available
by sholanozie on Saturday July 28, 2012 @05:46PM
(#4306610)
The retrieval and processing of the data is done externally (i.e. not on your server).
[ Parent
| on HN ]
Re:US Census API now available
by yummyfajitas on Saturday July 28, 2012 @06:09PM
(#4306672)
CA is 973M as zipped csv. CA is a bit over 10% of the US population, so the whole data set will be about 10gb. You can fit that on one of the cheaper linodes pretty easily.
With a few indices and maybe even a materialized view (or even pruning data you don't need), you can answer most queries so fast that latency between linode and the census > query time.
I know munging csv's and writing sql isn't as sexy as JSON APIs or mongodb, but sometimes the simple solutions are the right ones.
>CA is 973M as zipped csv. CA is a bit over 10% of the US population, so the whole data set will be about 10gb. You can fit that on one of the cheaper linodes pretty easily.
You forgot about unzipping. The total is 13.49 GB zipped, or 133.4 GB unzipped. [0]
[0] http://www2.census.gov/census_2010/04-Summary_File_1/Califor...
Re:US Census API now available
by sholanozie on Saturday July 28, 2012 @07:55PM
(#4306980)
I don't disagree with you, but for many people it's far more convenient to query an external API than to create their own.
Why should I have to install Postgres to be able to play around with data? This is one less step to think about.
Re:US Census API now available
by yummyfajitas on Sunday July 29, 2012 @12:13AM
(#4307505)
As I pointed out in my first comment, you can also just load it with pandas/excel/etc. I brought up postgres for the specific case of "queries on-demand on your web server or mobile app".