News for hackers. Stuff that matters.

Ask HN: Web crawler/data product ideas?

Is there a data set you would love to have but you just don't have time to compile?

Is there a data set that you currently compile manually that you can't live without?

Would it make your life easier if you could avoid opening all those browser tabs every day/week/month and instead purchase a neat file (csv/google spreadsheet/something else?) with everything you need? Which browser tabs would this eliminate?

( 8 comments | 9 points | on HN )
Re:Ask HN: Web crawler/data product ideas? by JamieLewis on Tuesday January 07, 2014 @11:42PM (#7021732)
Probably not what you are looking for, but I would love to see more real-time data curation - I would certainly be willing to hand over cash for some decently priced real time data feeds - stock tickers, sports events, raw traffic data, headlines, currency, flights - many of these are available in pull formats, but it would be great to get a push feed.
Re:Ask HN: Web crawler/data product ideas? by MechanicalTwerk on Wednesday January 08, 2014 @12:16AM (#7021836)
Any good examples of a push feed you currently like?
Re:Ask HN: Web crawler/data product ideas? by JamieLewis on Wednesday January 08, 2014 @12:29AM (#7021889)
Twitters streaming API used to work fairly well when I was playing with it. https://dev.twitter.com/docs/streaming-apis that was a few years ago though.
Re:Ask HN: Web crawler/data product ideas? by dalek2point3 on Wednesday January 08, 2014 @12:09AM (#7021815)
one big market for you is researchers. if only i had a penny for each time some economist came to me asking for a list of xxx in yyy, where x could be anything from flight prices, to businesses, to startups and yyy could be city / state/ coutnry /world. The internet is really good for current snapshots, but what reserachers most often need is repeated snapshots of the same thing. that helps us analyze how things change over itme. if you're a data company, please, please dont overwrite your tables to "update". save the old data. its really really useful. for research. and more importantly, for you.
Re:Ask HN: Web crawler/data product ideas? by MechanicalTwerk on Wednesday January 08, 2014 @12:15AM (#7021831)
Thanks. Will definitely look deeper into this. Know of any good sites where researchers make such requests?
Re:Ask HN: Web crawler/data product ideas? by bobosha on Wednesday January 08, 2014 @12:22AM (#7021860)
Have you looked at http://enigma.io/? They won TC Disrupt last year
Re:Ask HN: Web crawler/data product ideas? by Dnguyen on Tuesday January 07, 2014 @10:57PM (#7021583)
Priconomic just refocus to this business model
Re:Ask HN: Web crawler/data product ideas? by MechanicalTwerk on Tuesday January 07, 2014 @11:04PM (#7021609)
Priceonomics looks like more of a professional service at the moment. I'd like to sell actual data sets in more of an ecommerce/subscription model. Kind of like http://www.cbinsights.com/datastore/