News for hackers. Stuff that matters.
Re:Ask HN: Has Google/Bing won the search engine war? by krishna2 on Monday March 31, 2014 @07:57PM (#7505255)
Have you tried blekko.com ? The slashtags feature specifically does what you are describing here.

Of course, using this method will not get you data from sites that explicitly ban crawling (like yelp.com).

Disclaimer: I used to work at blekko and I implemented the slashtags feature.

Re:Ask HN: Has Google/Bing won the search engine war? by greglindahl on Monday March 31, 2014 @08:08PM (#7505307)
blekko's "ScoutJet" crawler is allowed to crawl yelp. We sent them an email before we launched, and they were nice enough to allow it.
Re:Ask HN: Has Google/Bing won the search engine war? by palakchokshi on Tuesday April 01, 2014 @01:34AM (#7506341)
I see ScoutJet in Yelp's Robots.txt file. I guess I'll have to ask them nicely to allow my crawler too. But Yelp is one of many sites that don't allow crawlers so I'll have to ask each one nicely to allow mine. I had even set up a temp google site explaining what my crawler uses the data for (essentially showing results with correct attribution and driving traffic back to original site).
Re:Ask HN: Has Google/Bing won the search engine war? by palakchokshi on Tuesday April 01, 2014 @01:55AM (#7506415)
Do you know if Blekko asked all the sites it wanted to crawl, to allow its ScoutJet crawler to crawl their pages?