I always ask there if I can't find what I'm looking for.
Here are more and more data sets. These are general data sets. Email me if you have a specific data set in mind (e.g. web-as-corpus, spam, images, social, reviews, etc.). I have a big file of information.
Good instructions:
http://corpus.leeds.ac.uk/internet.html#description
http://sslmit.unibo.it/~baroni/bootcat.html
http://www.drni.de/wac-tk/index.php/Documentation
Re:Free, Public Data Sets
by zipdog on Tuesday February 01, 2011 @10:15AM
(#2165737)
The wikipedia dump is great, but I've started using http://wiki.dbpedia.org/ which has an API to query the dumps.
Thanks for these, iisbum. I wish more public data was available in db, xml or similar structures - too often I find myself scraping government sites or pdfs to get the tables I need
Re:Free, Public Data Sets
by hessenwolf on Wednesday February 02, 2011 @05:03AM
(#2169717)
I looked at the site, and I see some data but I didn't find what I would have hoped for. I couldn't find yield curves, and historical exchange rates <i> up to <i/> today (available on the ecb site in xml format). Certainly I would have thought yield curves were a front page item.
Things that would be very cool would be 1. financial statements in a database format. I know you can scrape this but I don't know if they are available legitimately?
2. Historial Implied volatilities and historical observed volatilities.
Re:Free, Public Data Sets
by casperc on Tuesday February 01, 2011 @04:21PM
(#2167458)
Their write that most the data is available for download. I can't find it anywhere though, only the various APIs. Have they remove the possibility of downloading the data?
Re:Free, Public Data Sets
by pwenzel on Tuesday February 01, 2011 @03:50PM
(#2167316)
For those interested in transit data, check out the GTFS Data Exchange, a directory of many agencies' scheduling and map data, following the Google Transit Feed Specification.
Re:Free, Public Data Sets
by nico_h on Tuesday February 01, 2011 @05:40PM
(#2167785)
http://www.naturalearthdata.com/
From the website : Natural Earth is a public domain map dataset available at 1:10m, 1:50m, and 1:110 million scales as tightly integrated vector and raster data ...
Re:Free, Public Data Sets
by WildUtah on Tuesday February 01, 2011 @03:39PM
(#2167259)
Does anybody have precinct-level election results for the USA? A set for recent elections would be great for public access redistricting apps that will become relevant this year.
Re:Free, Public Data Sets
by toisanji on Tuesday February 01, 2011 @05:16PM
(#2167686)
anyone know of a dataset that has dates for when companies when companies registered or announced in the news? For example I would like to see the data hackernews was launched.
Re:Free, Public Data Sets
by jmtame on Tuesday February 01, 2011 @01:07PM
(#2166492)
i've had trouble finding geographical boundaries on neighborhoods in U.S. cities (e.g. downtown areas and residential neighborhoods). anyone know where i can find this?
Re:Free, Public Data Sets
by kfranken on Tuesday February 01, 2011 @06:02PM
(#2167879)
It's not exactly neighborhoods, but the US Census TIGER database has block and blockgroup boundaries with associated demographic data. You could probably synthesize that into "neighborhood" definitions. http://www.census.gov/geo/www/tiger/tgrshp2010/tgrshp2010.ht...
Re:Free, Public Data Sets
by _topher on Tuesday February 01, 2011 @01:22PM
(#2166564)
Thank you all for posting links and links to links to datasets, I have an unrelenting interest in data aggregation and machine learning, and didn't even know where to start. So helpful, and I am no longer stuck. :)
Re:Free, Public Data Sets
by fedd on Tuesday February 01, 2011 @01:38PM
(#2166638)
do all of them have some uniformed api? that would be great, ideally. query and cache all of them on demand from your own app without additional programming.
http://datasets.reddit.com
http://opendata.reddit.com
and
http://www.quora.com/Where-can-I-get-large-datasets-open-to-...
for some good lists of available stuff.
I always ask there if I can't find what I'm looking for.
Here are more and more data sets. These are general data sets. Email me if you have a specific data set in mind (e.g. web-as-corpus, spam, images, social, reviews, etc.). I have a big file of information.
Web as corpus: etc. Email me if you need more http://cleaneval.sigwac.org.uk/ http://liste.sslmit.unibo.it/pipermail/sigwac/2007-November/... http://wacky.sslmit.unibo.it/doku.php?id= http://clic.cimec.unitn.it/marco/research.htmlhttp://radar.oreilly.com/2010/03/open-data-pointers.html
http://www.datawrangling.com/some-datasets-available-on-the-...
http://del.icio.us/pskomoroch/dataset
http://infochimps.com/collections/datamob (and the other collections on the site)
http://www.data.gov/
Thanks for these, iisbum. I wish more public data was available in db, xml or similar structures - too often I find myself scraping government sites or pdfs to get the tables I need
If you're up to something in the economic data space we'd love to talk. Happy to take this to email ([email protected]) if anyone's interested.
Things that would be very cool would be 1. financial statements in a database format. I know you can scrape this but I don't know if they are available legitimately? 2. Historial Implied volatilities and historical observed volatilities.
Is it your site? Are you going to add yield curves?
http://lib.stat.cmu.edu/datasets/
http://datamob.org/
http://code.google.com/p/google-refine/
The Comprehensive Knowledge Archive Network! Pretty sweet resource really.
http://www.factual.com/
Edit: Although the use of this database is not free, I believe for personal use is just fine to download and experiment...
http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=...
This data set, contributed by Google Inc., contains English word n-grams and their observed frequency counts.
http://baseball1.com/statistics/
Some free, some paid.
http://www.gtfs-data-exchange.com/
http://unstats.un.org
some free, some paid
http://infochimps.com/
AIS Data (Marine Traffic)
http://www.aishub.net/
http://www.marinetraffic.com/ais/
And there's a great list of sources on Quora
http://www.quora.com/Where-can-I-get-large-datasets-open-to-...
Geonames : http://download.geonames.org/export/dump/
OS Open Data (UK Specific) : http://www.ordnancesurvey.co.uk/oswebsite/opendata/
http://data.govloop.com/ has data and lots of pointers to local government data.
Also I'm surprised no one mentioned Carl Malamud's site: http://public.resource.org/ - Lots of US gov't and legal data in friendly formats.
https://github.com/mohitranka/TwitterSentimentCorpora
https://www.cia.gov/library/publications/download/
bookmarked and shared this thread.
Something like satellite imagery, medical imaging, semiconductor masks and wafers photos or CAD files, etc.
Any pointers?
Some other applications, example CT Colonography http://www.acrin.org/
Anyone know of any publicly available song lyric databases?
All the datasets I found on www, are Huge (in double digit GBs..).