Re:Ask HN: Has Google/Bing won the search engine war?
by palakchokshi on Monday March 31, 2014 @02:26PM
(#7503220)
http://yelp.com/robots.txt
It seems you can contact Yelp and tell them how you plan to use their data and maybe they'll let you crawl their site.
I really want to explore any and all alternative options before I decide to crawl away :)
Re:Ask HN: Has Google/Bing won the search engine war?
by argumentum on Monday March 31, 2014 @04:20PM
(#7503855)
Are you hacker, or aren't you? Dumb rules exist to be broken.
Re:Ask HN: Has Google/Bing won the search engine war?
by palakchokshi on Monday March 31, 2014 @04:26PM
(#7503901)
Agreed but no harm in accessing the vast knowledge of the HN community to exhaust alternatives before tightening my hacker cap and plunging in head first.
Re:Ask HN: Has Google/Bing won the search engine war?
by palakchokshi on Monday March 31, 2014 @02:35PM
(#7503270)
Yelp is going after startups that crawl their site[1]. I don't want to make Yelp the poster site for this because other big sites do this too.
[1]http://www.courthousenews.com/2012/01/27/43403.htm
Permitting certain search engines to crawl but not others is anticompetitive and violates the principle of an open web.
> any new search engine will have its roots in "shady" tactics of crawling.
Who cares? If you are successful enough, then you'll negotiate with them later. Don't worry about it now.
Crawl away!