News for hackers. Stuff that matters.

A comment on Ask HN: Is there room for another search engine?. Parent comment

Re:Ask HN: Is there room for another search engine? by thesmallestcat on Sunday March 26, 2017 @01:59PM (#13962081)
If Google today started ignoring robots.txt, not many people would start blocking Google's crawlers, assuming they continued to do their job efficiently. robots.txt is security by obscurity at best.
[ Parent | on HN ]
Re:Ask HN: Is there room for another search engine? by Analemma_ on Sunday March 26, 2017 @02:17PM (#13962185)
People wouldn't block Google because allowing Google scraping offers a return in the form of more traffic that offsets the cost. $BRAND_NEW_ENGINE wouldn't have that advantage.
[ Parent | on HN ]
Re:Ask HN: Is there room for another search engine? by thesmallestcat on Sunday March 26, 2017 @02:23PM (#13962224)
Google's crawlers don't exactly perform 2FA before they crawl a website. If impersonating Google's crawlers doesn't suit your fancy, there are all manner of ways to anonymize a crawler. So I think blocking is one of the least interesting challenges. The other side of the coin is robots.txt is not inherently adversarial, and a crawler could waste quite some time and energy crawling truly meaningless content. That, in my mind, is the interesting challenge: Obsoleting robots.txt with an intelligent (and gentle) crawler.
[ Parent | on HN ]
Re:Ask HN: Is there room for another search engine? by desdiv on Sunday March 26, 2017 @03:03PM (#13962475)
Google doesn't feel like being impersonated so they offer an easy way to authenticate Googlebot:

https://support.google.com/webmasters/answer/80553?hl=en

[ Parent | on HN ]
Re:Ask HN: Is there room for another search engine? by angry-hacker on Monday March 27, 2017 @06:06AM (#13966365)
They do ignore robots.txt sometimes, the pages just font end up in serps.
[ Parent | on HN ]