News for hackers. Stuff that matters.
Re:Where can I get large datasets open to the public? by physcab on Tuesday April 05, 2011 @12:41AM (#2409733)
Asking "What datasets are available to me?" is sometimes the wrong question. A better way of going about the problem is asking something more specific like "How can I create a heat-map of U.S poverty?" The reason why the latter is better is that it not only focuses your attention on something do-able but it actually teaches you more about data analysis than just searching for datasets.

For example, to solve the question above you are going to be asking yourself the following followup questions:

1) Where do I get a map of the U.S?

2) How do I make a heat-map?

3) How do I feed in my own data into this heat map?

4) What colors do I use?

5) Can I do this real-time? Do I need a database? What language do I use?

6) Whats a FIPS code?

7) How do I find a poverty dataset with FIPS codes?

8) This poverty dataset doesn't have FIPS codes, but I can join it with this other dataset that does have FIPS codes.

Re:Where can I get large datasets open to the public? by buddydvd on Tuesday April 05, 2011 @03:36AM (#2410071)
Open datasets are hard to come by. It's potentially easier to find problems to solve by looking at the available datasets than seeking datasets for the problems you wish to solve.
Re:Where can I get large datasets open to the public? by physcab on Tuesday April 05, 2011 @11:16AM (#2411241)
When you want to create a new website do you start by looking at a bunch of clip art and images? Browsing through datasets such as the ones listed in this thread leaves me overwhelmed. I've never had any difficulty finding open datasets. If there is a dataset I need that doesn't exist or costs money, I find a way to create it from scratch.
Re:Where can I get large datasets open to the public? by spiffytech on Tuesday April 05, 2011 @11:14PM (#2413769)
Datasets aren't always quested for. Sometimes you want a specific dataset to solve a specific problem, and sometimes you'll take any dataset just to see what you can learn, or to make interesting infographics. Lists like this help the latter situation.
Re:Where can I get large datasets open to the public? by epo on Tuesday April 05, 2011 @08:58AM (#2410721)
It may sometimes be the right question, it all depends on what the enquirer wants to with the data. This comment is just an exercise in answering a question that no one asked, I fail to see why it merited any upvotes.
Re:Where can I get large datasets open to the public? by physcab on Tuesday April 05, 2011 @11:22AM (#2411269)
It is just an example of an alternate viewpoint. My point is that sometimes you will have more success finding data for questions that interest you than looking at datasets that interest you and trying to come up with questions.
Re:Where can I get large datasets open to the public? by lars512 on Tuesday April 05, 2011 @02:22AM (#2409938)
I likewise feel that this type of list of data sources fits a data collecting/hoarding mentality, rather than a problem solving one. It doesn't help me think of interesting things to do with the data.