News for hackers. Stuff that matters.
Re:Ask HN: What data set would you like to see/use, but does... by arthurdent on Saturday July 03, 2010 @10:44PM (#1484998)
1. open financial tick data -- opentick used to have some, but that's gone as far as i can tell. also, one of the biggest problems with financial data is cleaning it. with an open dataset, people might be encouraged to clean the open dataset like people contribute patches to open source code.

2. open restaurant data -- comprehensive listings of restaurants with information like hours, address, possibly menus.

Re:Ask HN: What data set would you like to see/use, but does... by transatlantic on Sunday July 04, 2010 @01:44AM (#1485232)
I've spent a non-trivial amount of effort trying to figure out how to do number two at scale, with a high level of accuracy, in a manner that could eventually be profitable. So far I'm not close.
Re:Ask HN: What data set would you like to see/use, but does... by arthurdent on Sunday July 04, 2010 @02:14AM (#1485251)
I would be interested in finding out more about your approach to this or chatting about ways to make this happen.

I found this old thread: http://news.ycombinator.com/item?id=911853 but couldn't find your contact information.

Re:Ask HN: What data set would you like to see/use, but does... by buro9 on Sunday July 04, 2010 @04:37AM (#1485386)
Over at Yell we have such data and I'm trying to persuade them to open it up through a public facing API.

Restaurants, locations, contacts details, opening times, photos of the facade and sometimes of the inside, some menus.

We also have a lot of other data for things like bars (what beers they have, do they have a garden, how many TV screens) and other business types.

The obstacles we've encountered to opening the data are: 1) Fear. Parts of Yell fear that the data they spend time and money creating and collating will be scraped and the crown jewels given away for free. 2) Politics. Different parts of Yell view the opening of data differently and will fight over any move even if they are in general agreement... nothing gets done.

I am trying to argue that on #1 that if they really need to fear something it should be not moving with the times. And also through the use of things like API keys that we could rate limit to some extent to prevent major scraping occurring if they really think it's a big problem.

And on #2 it's currently part of their business to argue a lot it seems.

I believe that if they could see the value in opening data that both would be a non-issue and it would just be done. So I'm also working on trying to document examples of how people could use such an API that is beneficial to Yell. And on that note... if anyone has such examples I'd be glad to hear them.

Re:Ask HN: What data set would you like to see/use, but does... by steveitis on Sunday July 04, 2010 @12:54PM (#1485928)
It's pretty simple. I've never heard of Yell before now. If they'd opened up this data, than I would have. Especially if they only allowed usage under an attribution required license.
Re:Ask HN: What data set would you like to see/use, but does... by aw3c2 on Sunday July 04, 2010 @05:00AM (#1485415)
Open restaurant data is slowly added into OpenStreetMap. I too think it is a great resource (or will be), same for shops, doctors etc.
Re:Ask HN: What data set would you like to see/use, but does... by retube on Sunday July 04, 2010 @01:00PM (#1485938)
Your number 2 is interesting. My company actually basically does this. We crawl the UK web for business sites, incl. retail and hospitality/leisure and suck out contact details, addresses, products, services, food & drink items on menus, opening times as well as a few other things. We're looking at making available an API.
Re:Ask HN: What data set would you like to see/use, but does... by revorad on Sunday July 04, 2010 @05:24AM (#1485426)
1. The R Empirical Finance Taks View lists a number of packages with data - http://cran.r-project.org/web/views/Finance.html. Especially check out quantmod- http://www.quantmod.com.
Re:Ask HN: What data set would you like to see/use, but does... by ig1 on Sunday July 04, 2010 @07:32AM (#1485527)
Cleaning financial tick data can be very tricky, because it depends very much on how you want to use the data for. If you're not careful any models you build on the cleaned data can fall apart in real world usage because they fail to take into account the abnormalities in market data.