Re:Ask HN: Is there room for another search engine?
by Analemma_ on Sunday March 26, 2017 @02:17PM
(#13962185)
People wouldn't block Google because allowing Google scraping offers a return in the form of more traffic that offsets the cost. $BRAND_NEW_ENGINE wouldn't have that advantage.
[ Parent
| on HN ]
Re:Ask HN: Is there room for another search engine?
by thesmallestcat on Sunday March 26, 2017 @02:23PM
(#13962224)
Google's crawlers don't exactly perform 2FA before they crawl a website. If impersonating Google's crawlers doesn't suit your fancy, there are all manner of ways to anonymize a crawler. So I think blocking is one of the least interesting challenges. The other side of the coin is robots.txt is not inherently adversarial, and a crawler could waste quite some time and energy crawling truly meaningless content. That, in my mind, is the interesting challenge: Obsoleting robots.txt with an intelligent (and gentle) crawler.