News for hackers. Stuff that matters.
Re:Ask HN: Has Google/Bing won the search engine war? by timr on Monday March 31, 2014 @02:21PM (#7503188)
DDG uses Bing's search API. That's why you see Yelp on DDG.

I used to work at Yelp, and I can tell you that if there were no restrictions on who crawled the site, we'd have been regularly DOSed by robots. I don't (and didn't) like the precedent, but I understand some of the motivation.

Re:Ask HN: Has Google/Bing won the search engine war? by palakchokshi on Monday March 31, 2014 @02:31PM (#7503244)
I understand the motivation too. My request is that there should me some option for crawlers to apply to be whitelisted. I know Yelp asks you to contact them if you want to crawl them and they'll decide on a per case basis. It would be nice if there were standards that sites could implement to whitelist crawlers, create crawl schedules for whitelisted crawlers, something on the lines of robots.txt but with attributes that told a specific crawler when and how frequently they can crawl.
Re:Ask HN: Has Google/Bing won the search engine war? by timr on Monday March 31, 2014 @02:34PM (#7503264)
It wouldn't work. The crawlers that cause the problems are not the ones that pay attention to voluntary management systems like robots.txt.
Re:Ask HN: Has Google/Bing won the search engine war? by palakchokshi on Monday March 31, 2014 @02:42PM (#7503314)
So a "legit" crawler has no recourse but to resort to implementing technology that'll circumvent anti-crawler code? Can't say I didn't try to go legit :)
Re:Ask HN: Has Google/Bing won the search engine war? by timr on Tuesday April 01, 2014 @03:40AM (#7506688)
Well, no: you ask, and if they say no, you don't get to crawl. The rules don't change because you think they're unjust.

If they catch you crawling, they'll send you a cease and desist letter, so do it at your own risk.

Re:Ask HN: Has Google/Bing won the search engine war? by palakchokshi on Tuesday April 01, 2014 @12:22PM (#7509446)
Agreed. I would rather not go down a route of building my product and then have a bunch of cease and desist orders taking away a lot of my search results. Asking nicely is the next step for me.