News for hackers. Stuff that matters.
Re:Free, Public Data Sets by mindcrime on Tuesday February 01, 2011 @09:26AM (#2165551)
If anyone is looking for more datasets, see:

http://datasets.reddit.com

http://opendata.reddit.com

and

http://www.quora.com/Where-can-I-get-large-datasets-open-to-...

for some good lists of available stuff.

Re:Free, Public Data Sets by bravura on Tuesday February 01, 2011 @02:42PM (#2166986)
get.theinfo is the best way to find data sets. They are a bunch of data hoarders who can help you: http://groups.google.com/group/get-theinfo/?pli=1

I always ask there if I can't find what I'm looking for.

Here are more and more data sets. These are general data sets. Email me if you have a specific data set in mind (e.g. web-as-corpus, spam, images, social, reviews, etc.). I have a big file of information.

    http://theinfo.org/
    http://infochimps.org/datasets
    http://ckan.org [Comprehensive Knowledge Archive Network]
    http://www.datawrangling.com/some-datasets-available-on-the-web.html
    http://del.icio.us/pskomoroch/dataset
    http://www.reddit.com/r/datasets/
    http://news.ycombinator.com/item?id=1242029
    http://www.reddit.com/r/opendata
    http://www.trustlet.org/wiki/Repositories_of_datasets
    http://www.daniel-lemire.com/blog/data-for-data-mining/
    http://www.quantlet.org/mdbase/
    http://datamob.org/
    http://freebase.com/
    http://infochimp.info/ics/data/ripd/www-personal.umich.edu/~mejn/netdata/
    http://www.archive-it.org/public/all_collections

    Large:
        http://www.ckan.net/tag/read/size-large
        http://www.diggingintodata.org/Repositories/tabid/167/Default.aspx
Web as corpus:

    Good instructions:
        http://corpus.leeds.ac.uk/internet.html#description
    http://sslmit.unibo.it/~baroni/bootcat.html

    http://www.drni.de/wac-tk/index.php/Documentation
etc. Email me if you need more http://cleaneval.sigwac.org.uk/ http://liste.sslmit.unibo.it/pipermail/sigwac/2007-November/... http://wacky.sslmit.unibo.it/doku.php?id= http://clic.cimec.unitn.it/marco/research.html
Re:Free, Public Data Sets by zipdog on Tuesday February 01, 2011 @10:15AM (#2165737)
The wikipedia dump is great, but I've started using http://wiki.dbpedia.org/ which has an API to query the dumps.

Thanks for these, iisbum. I wish more public data was available in db, xml or similar structures - too often I find myself scraping government sites or pdfs to get the tables I need

Re:Free, Public Data Sets by adw on Tuesday February 01, 2011 @09:46AM (#2165631)
We've got quite a lot of public economic data: http://timetric.com/.

If you're up to something in the economic data space we'd love to talk. Happy to take this to email ([email protected]) if anyone's interested.

Re:Free, Public Data Sets by hessenwolf on Wednesday February 02, 2011 @05:03AM (#2169717)
I looked at the site, and I see some data but I didn't find what I would have hoped for. I couldn't find yield curves, and historical exchange rates <i> up to <i/> today (available on the ecb site in xml format). Certainly I would have thought yield curves were a front page item.

Things that would be very cool would be 1. financial statements in a database format. I know you can scrape this but I don't know if they are available legitimately? 2. Historial Implied volatilities and historical observed volatilities.

Re:Free, Public Data Sets by adw on Tuesday February 08, 2011 @05:32AM (#2192506)
http://timetric.com/dataset/exchange_rates_forex_europe/ for the exchange-rate data, at least.
Re:Free, Public Data Sets by hessenwolf on Tuesday February 08, 2011 @01:00PM (#2193944)
Okay - it's there...

Is it your site? Are you going to add yield curves?

Re:Free, Public Data Sets by gtani on Tuesday February 01, 2011 @11:24AM (#2166031)
Re:Free, Public Data Sets by sosuke on Tuesday February 01, 2011 @10:34AM (#2165811)
Heh, a day after he leaves HN he makes the first page. He will still be here whether he visits the site or not.
Re:Free, Public Data Sets by cstuder on Tuesday February 01, 2011 @12:04PM (#2166237)
And I recently discovered Google Refine, for cleaning up messy datasets.

http://code.google.com/p/google-refine/

Re:Free, Public Data Sets by LiveTheDream on Tuesday February 01, 2011 @03:23PM (#2167182)
Re:Free, Public Data Sets by agentultra on Tuesday February 01, 2011 @09:48AM (#2165639)
What about http://ckan.org/ ?

The Comprehensive Knowledge Archive Network! Pretty sweet resource really.

Re:Free, Public Data Sets by djsun on Friday February 04, 2011 @04:09PM (#2181241)
The CKAN software is a platform for hosting data and metadata, but as far as I see, http://ckan.org does not actually list data sets.
Re:Free, Public Data Sets by pudo on Sunday February 06, 2011 @06:18AM (#2185411)
try http://ckan.net for the data, http://ckan.org is for the software behind it :)
Re:Free, Public Data Sets by dmpayton on Tuesday February 01, 2011 @11:47AM (#2166152)
Kinda surprised no one has mentioned Factual. I'm using some of their diabetes data for my side-startup.

http://www.factual.com/

Re:Free, Public Data Sets by casperc on Tuesday February 01, 2011 @04:21PM (#2167458)
Their write that most the data is available for download. I can't find it anywhere though, only the various APIs. Have they remove the possibility of downloading the data?
Re:Free, Public Data Sets by steveklabnik on Tuesday February 01, 2011 @11:11AM (#2165967)
Don't forget Stack Overflow! http://data.stackexchange.com/
Re:Free, Public Data Sets by svag on Tuesday February 01, 2011 @11:50AM (#2166172)
There is also the IMDB database in various format provided by IMDB itself here: http://www.imdb.com/interfaces

Edit: Although the use of this database is not free, I believe for personal use is just fine to download and experiment...

Re:Free, Public Data Sets by agbell on Tuesday February 01, 2011 @02:02PM (#2166764)
Non-Free Google data:

http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=...

This data set, contributed by Google Inc., contains English word n-grams and their observed frequency counts.

Re:Free, Public Data Sets by hvs on Tuesday February 01, 2011 @12:25PM (#2166336)
Don't forget the Lahman Baseball Database with information from 1871-2010

http://baseball1.com/statistics/

Re:Free, Public Data Sets by jamwt on Tuesday February 01, 2011 @01:19PM (#2166547)
And, for very detailed play-by-play data for decades of games, check out retrosheet: http://www.retrosheet.org/game.htm
Re:Free, Public Data Sets by balakk on Tuesday February 01, 2011 @10:34AM (#2165814)
https://datamarket.azure.com/

Some free, some paid.

Re:Free, Public Data Sets by damoncali on Tuesday February 01, 2011 @10:27AM (#2165780)
http://infochimps.com also has a bunch.
Re:Free, Public Data Sets by pwenzel on Tuesday February 01, 2011 @03:50PM (#2167316)
For those interested in transit data, check out the GTFS Data Exchange, a directory of many agencies' scheduling and map data, following the Google Transit Feed Specification.

http://www.gtfs-data-exchange.com/

Re:Free, Public Data Sets by tszming on Tuesday February 01, 2011 @10:46AM (#2165872)
Open Directory RDF Dump: http://rdf.dmoz.org/
Re:Free, Public Data Sets by Perceval on Tuesday February 01, 2011 @01:06PM (#2166489)
For international relations data, Correlates of War hosts a number of data sets: http://www.correlatesofwar.org/Datasets.htm
Re:Free, Public Data Sets by joubert on Tuesday February 01, 2011 @10:46AM (#2165866)
I have links to a few govt.-provided data sets at http://elev.at
Re:Free, Public Data Sets by jcr on Tuesday February 01, 2011 @10:38PM (#2168857)
United Nations stats (lots of goodies)

http://unstats.un.org

some free, some paid

http://infochimps.com/

AIS Data (Marine Traffic)

http://www.aishub.net/

http://www.marinetraffic.com/ais/

And there's a great list of sources on Quora

http://www.quora.com/Where-can-I-get-large-datasets-open-to-...

Re:Free, Public Data Sets by llimllib on Tuesday February 01, 2011 @11:12AM (#2165972)
Re:Free, Public Data Sets by l0nwlf on Wednesday February 02, 2011 @04:33AM (#2169651)
Re:Free, Public Data Sets by LiveTheDream on Tuesday February 01, 2011 @03:21PM (#2167170)
I track datasets that I come across at http://www.delicious.com/tobym/dataset
Re:Free, Public Data Sets by lkozma on Tuesday February 01, 2011 @02:57PM (#2167069)
Re:Free, Public Data Sets by nico_h on Tuesday February 01, 2011 @05:40PM (#2167785)
http://www.naturalearthdata.com/ From the website : Natural Earth is a public domain map dataset available at 1:10m, 1:50m, and 1:110 million scales as tightly integrated vector and raster data ...
Re:Free, Public Data Sets by WildUtah on Tuesday February 01, 2011 @03:39PM (#2167259)
Does anybody have precinct-level election results for the USA? A set for recent elections would be great for public access redistricting apps that will become relevant this year.
Re:Free, Public Data Sets by toisanji on Tuesday February 01, 2011 @05:16PM (#2167686)
anyone know of a dataset that has dates for when companies when companies registered or announced in the news? For example I would like to see the data hackernews was launched.
Re:Free, Public Data Sets by jmtame on Tuesday February 01, 2011 @01:07PM (#2166492)
i've had trouble finding geographical boundaries on neighborhoods in U.S. cities (e.g. downtown areas and residential neighborhoods). anyone know where i can find this?
Re:Free, Public Data Sets by kfranken on Tuesday February 01, 2011 @06:02PM (#2167879)
It's not exactly neighborhoods, but the US Census TIGER database has block and blockgroup boundaries with associated demographic data. You could probably synthesize that into "neighborhood" definitions. http://www.census.gov/geo/www/tiger/tgrshp2010/tgrshp2010.ht...
Re:Free, Public Data Sets by timr on Tuesday February 01, 2011 @06:06PM (#2167889)
Re:Free, Public Data Sets by ericwaller on Tuesday February 01, 2011 @04:04PM (#2167387)
It's not in dump format, but you should take a look at simplegeo's (free) api: http://simplegeo.com/
Re:Free, Public Data Sets by eli on Tuesday February 01, 2011 @11:59PM (#2169053)
Some US Gov't data sites no one else mentioned:

http://data.govloop.com/ has data and lots of pointers to local government data.

Also I'm surprised no one mentioned Carl Malamud's site: http://public.resource.org/ - Lots of US gov't and legal data in friendly formats.

Re:Free, Public Data Sets by mcauser on Tuesday February 01, 2011 @05:47PM (#2167823)
Heaps of useful info: http://www.nationmaster.com
Re:Free, Public Data Sets by random42 on Thursday February 03, 2011 @04:49AM (#2174238)
I'd prepared (based on other datasets) a smallish movie tweet dataset. You may find it useful, if working with tweets and/or reviews.

https://github.com/mohitranka/TwitterSentimentCorpora

Re:Free, Public Data Sets by youknow on Wednesday February 02, 2011 @03:56AM (#2169592)
CIA World Factbook (demographics, geography, communications, government, economy, military stats of countries):

https://www.cia.gov/library/publications/download/

Re:Free, Public Data Sets by _topher on Tuesday February 01, 2011 @01:22PM (#2166564)
Thank you all for posting links and links to links to datasets, I have an unrelenting interest in data aggregation and machine learning, and didn't even know where to start. So helpful, and I am no longer stuck. :)
Re:Free, Public Data Sets by fedd on Tuesday February 01, 2011 @01:38PM (#2166638)
do all of them have some uniformed api? that would be great, ideally. query and cache all of them on demand from your own app without additional programming.

bookmarked and shared this thread.

Re:Free, Public Data Sets by nivertech on Wednesday February 02, 2011 @05:44AM (#2169784)
I looking for free public domain large high-resolution imaging datasets.

Something like satellite imagery, medical imaging, semiconductor masks and wafers photos or CAD files, etc.

Any pointers?

Re:Free, Public Data Sets by brainid on Wednesday February 02, 2011 @07:43AM (#2169996)
Here are medical imaging datasets I am aware of: Neuroimaging (see http://www.nitrc.org for others) OASIS http://www.oasis-brains.org/ ADNI http://adni.loni.ucla.edu/ (huge dataset, requires application) OpenfMRI http://openfmri.org/ EEG http://eeg.pl/epi

Some other applications, example CT Colonography http://www.acrin.org/

Re:Free, Public Data Sets by kaffeinecoma on Tuesday February 01, 2011 @02:17PM (#2166836)
This is a real treasure to come across. I hope we'll keep seeing jacquesm's blog postings here.

Anyone know of any publicly available song lyric databases?

Re:Free, Public Data Sets by random42 on Thursday February 03, 2011 @04:51AM (#2174242)
Anyplace I can find _small_ free web spam dataset? ( for commercial use, sorry :( )

All the datasets I found on www, are Huge (in double digit GBs..).

Re:Free, Public Data Sets by wladimir on Tuesday February 01, 2011 @12:52PM (#2166423)
Wow, useful stuff. This thread goes into my bookmarks.