News for hackers. Stuff that matters.

Ask HN: List of open/public databases

A year or two ago somebody posted a list of open/free/accessible datasources for hackers to download and play around with. I thought this was a great resource, so I saved it, but heck if I can find it now.

Does anybody have such a list? Things like zip-codes for the US, locations of Starbucks stores, current weather forecasts, list of major newspapers, list of publicly-traded stocks, etc. I know there are tons of open/free databases waiting for us to mashup, just can't seem to find a list of them.

EDIT: The goal is a downloadable chunk of data to mashup, reformat, and use. That means CSV/XML/etc format and a public/anonymous FTP or something.

( 26 comments | 64 points | on HN )
Re:Ask HN: List of open/public databases by randomtask on Wednesday July 07, 2010 @06:39AM (#1493856)
You might find the datasets/opendata sub-reddits interesting.

http://www.reddit.com/r/datasets http://www.reddit.com/r/opendata

Re:Ask HN: List of open/public databases by sidmitra on Wednesday July 07, 2010 @08:22AM (#1493952)
Delicious is a good place for the same. Eg.

http://delicious.com/sidmitra/datasets

Re:Ask HN: List of open/public databases by robin_reala on Wednesday July 07, 2010 @06:05AM (#1493812)
Freebase is a big one: http://www.freebase.com/ .

Also, if you’re in the UK the government datasets might interest you: http://data.gov.uk/

Re:Ask HN: List of open/public databases by IgorPartola on Wednesday July 07, 2010 @06:48AM (#1493863)
Re:Ask HN: List of open/public databases by brown9-2 on Wednesday July 07, 2010 @08:00AM (#1493931)
Re:Ask HN: List of open/public databases by DanielBMarkham on Wednesday July 07, 2010 @07:25AM (#1493897)
As I continue to research this today, I grow more and more amazed at the lack of public data sets. There are what? A million different websites that keep some kind of list of something? And I'm finding only a dozen or two databases publicly accessible? It's crazy. Everybody either wants to charge or actively wants to prevent you from getting their data -- even when the data is common knowledge, was gathered at public expense, etc.

Anybody from Google on? There has to be some shortcuts here somewhere.

Re:Ask HN: List of open/public databases by tszming on Wednesday July 07, 2010 @10:15AM (#1494238)
Amazon Public Data Sets: http://aws.amazon.com/publicdatasets/

It is fast and easy to use if you are already using EC2.

Re:Ask HN: List of open/public databases by zackham on Wednesday July 07, 2010 @11:12AM (#1494393)
How about weather - I have been looking around for an API to access historical weather data with no luck so far. Anyone familiar with the landscape of weather data options?
Re:Ask HN: List of open/public databases by joubert on Wednesday July 07, 2010 @12:16PM (#1494585)
Re:Ask HN: List of open/public databases by gspyrou on Wednesday July 07, 2010 @06:06AM (#1493816)
You may try taking a look at Microsoft Codename "Dallas" https://www.sqlazureservices.com/Catalog.aspx
Re:Ask HN: List of open/public databases by joubert on Wednesday July 07, 2010 @08:54AM (#1494002)
I have links to a few open (gov) data sets at the top of http://elev.at
Re:Ask HN: List of open/public databases by cullenking on Wednesday July 07, 2010 @04:00PM (#1495234)
We are running SRTM and NED elevation datasets, and have a web service (sinatra FTW!) exposing the data. I setup a variety of interpolation algorithms, which are fun to play with. If someone needs access to it and has a cool project, we are happy to provide it. Get ahold of me via email for info.

The SRTM data is < 20 gigs compressed, and I downloaded it from some source, can't find the link now. It was a SLOWWWWWW download, so took many days. Reminded me of my old 2400 baud modem.

the NED dataset is just shy of 500 gigs. We shipped a drive off to the USGS and they populated it and shipped it back. The 2 month wait sucked, but it was the only way to get it without paying a thousand bucks from a private company.

I am happy to share either the SRTM or the NED dataset we have, but sharing it is pretty much limited to shipping hard drives around :(

Re:Ask HN: List of open/public databases by iamelgringo on Wednesday July 07, 2010 @09:11AM (#1494042)
Re:Ask HN: List of open/public databases by truiu on Wednesday July 07, 2010 @09:14AM (#1494052)
The openstreetmap project lets you download its whole dataset: http://wiki.openstreetmap.org/wiki/Planet

They also got a API ... see the wiki.

Re:Ask HN: List of open/public databases by lpolovets on Wednesday July 07, 2010 @12:34PM (#1494623)
www.factual.com has a lot of data, much of which is freely downloadable. There's also an API for data access.

Examples:

zip codes in the US: http://www.factual.com/t/MwwkkU/US_Zip_Codes

stock symbols: http://www.factual.com/t/jyqdWC/market_symbols

I work at Factual, so if you have feature requests/comments/complaints please let me know! I'm leo at factual dot com

Re:Ask HN: List of open/public databases by vili on Wednesday July 07, 2010 @08:55AM (#1494007)
You can get all of Wikipedia here: http://en.wikipedia.org/wiki/Wikipedia_database
Re:Ask HN: List of open/public databases by what on Saturday July 10, 2010 @09:50AM (#1503642)
Re:Ask HN: List of open/public databases by lostbit on Wednesday July 07, 2010 @11:59AM (#1494533)
Lottery in Brazil (MegaSena) allow download of all results ever: http://www1.caixa.gov.br/loterias/loterias/megasena/download...
Re:Ask HN: List of open/public databases by Cherian on Thursday July 08, 2010 @08:20AM (#1497102)
Stackoverflow(includes sister sites) dump: http://blog.stackoverflow.com/2010/07/creative-commons-data-...
Re:Ask HN: List of open/public databases by warp on Wednesday July 07, 2010 @09:23AM (#1494087)
MusicBrainz has postgresql dumps available (partly public domain, partly cc-by-nc-sa):

http://musicbrainz.org/doc/MusicBrainz_Database

Re:Ask HN: List of open/public databases by cjoh on Wednesday July 07, 2010 @04:30PM (#1495325)
Re:Ask HN: List of open/public databases by tszming on Wednesday July 07, 2010 @10:11AM (#1494229)
Open Directory RDF Dump (e.g. Google Directory using it) http://rdf.dmoz.org/
Re:Ask HN: List of open/public databases by tomhaney on Wednesday July 07, 2010 @10:53AM (#1494341)
Re:Ask HN: List of open/public databases by binarymax on Wednesday July 07, 2010 @11:10AM (#1494391)
Re:Ask HN: List of open/public databases by kinlane on Wednesday July 07, 2010 @01:21PM (#1494767)
A list of several data sources and tools to work with the data: http://www.kinlane.com/data/