News for hackers. Stuff that matters.
Re:Ask HN: Has Google/Bing won the search engine war? by argumentum on Monday March 31, 2014 @02:12PM (#7503131)
> The issue is I want to respect their TOS.

Permitting certain search engines to crawl but not others is anticompetitive and violates the principle of an open web.

> any new search engine will have its roots in "shady" tactics of crawling.

Who cares? If you are successful enough, then you'll negotiate with them later. Don't worry about it now.

Crawl away!

Re:Ask HN: Has Google/Bing won the search engine war? by palakchokshi on Monday March 31, 2014 @02:26PM (#7503220)
http://yelp.com/robots.txt

It seems you can contact Yelp and tell them how you plan to use their data and maybe they'll let you crawl their site.

I really want to explore any and all alternative options before I decide to crawl away :)

Re:Ask HN: Has Google/Bing won the search engine war? by argumentum on Monday March 31, 2014 @04:20PM (#7503855)
Are you hacker, or aren't you? Dumb rules exist to be broken.
Re:Ask HN: Has Google/Bing won the search engine war? by palakchokshi on Monday March 31, 2014 @04:26PM (#7503901)
Agreed but no harm in accessing the vast knowledge of the HN community to exhaust alternatives before tightening my hacker cap and plunging in head first.
Re:Ask HN: Has Google/Bing won the search engine war? by palakchokshi on Monday March 31, 2014 @02:35PM (#7503270)
Yelp is going after startups that crawl their site[1]. I don't want to make Yelp the poster site for this because other big sites do this too. [1]http://www.courthousenews.com/2012/01/27/43403.htm
Re:Ask HN: Has Google/Bing won the search engine war? by AznHisoka on Monday March 31, 2014 @02:29PM (#7503234)
Agreed. 99% of the time these big companies don't care. Also, use proxies.

if they come knocking on your door to take down the content, then worry about it.