Re:Ask HN: Has Google/Bing won the search engine war?
by palakchokshi on Tuesday April 01, 2014 @01:34AM
(#7506341)
I see ScoutJet in Yelp's Robots.txt file. I guess I'll have to ask them nicely to allow my crawler too. But Yelp is one of many sites that don't allow crawlers so I'll have to ask each one nicely to allow mine. I had even set up a temp google site explaining what my crawler uses the data for (essentially showing results with correct attribution and driving traffic back to original site).
[ Parent
| on HN ]
There are no comments on this comment yet.