There were a surprising number of sites out there that explicitly blocked access to anyone but Google/Bing at the time.
We'd also get a dozen complaints or so a month from sites we'd crawled. Mostly upset about us using up their bandwidth, and telling us that only Google was allowed to crawl them (though having no robots.txt configured to say so).
If Google is taking traffic and reducing revenue, a company can deny in robots.txt. Google will actually follow those rules - unlike most others that are supposedly in this 2nd class.
The company did respect robots.txt, though it was initially a bit of a struggle to convince certain project managers to do so.
No. The internet is public. Publishers shouldn't get any say in who accesses their content or how they do it. As far as I'm concerned, the fact that they do is a bug.
Yes, the bad bots don't give a fuck, but even the non-malicious bots (ahrefs, moz, some university's search engine etc) don't bring any value to me as a site owner, take up band width and resources and fill up logs. If you can remove them with three lines in your robots.txt, that's less noise. Especially universities do, in my opinion, often behave badly and are uncooperative when you point out their throttling does not work and they're hammering your server. Giving them a "Go Away, You Are Not Wanted Here" in a robots.txt works for most, and the rest just gets blocked.
Why can't you just ratelimit IPs that are "too active" for your server to handle?
And yes, that means google gets special treatment.
Think about the model for a site like stackoverflow. The longest of long tail questions on that site: what’s the actual lifecycle of that question?
- posted by a random user - scraped by google, bing, et al - visited by someone who clicked on a search result on google - eventually, answered - hopefully, reindexed by google, bing et al - maybe never visited again because the answer now shows up on the google SERP
In the lifetime of that question how many times is it accessed by a human, compared to the number of times it’s requested and rerequested by an indexing bot?
What would be the impact on your site of three more bots as persistent as google bot? Why should you bother with their requests?
So yes, sites care about bot traffic and they care about google in particular.
Additional evidence here: https://knuckleheads.club/the-evidence-we-found-so-far/
They obviously don't care enough then - Google says you should use rdns to verify that googlebot crawls are real[0]. Cloudflare does this automatically now as well for customers with WAF (pro plan).
0: https://developers.google.com/search/docs/advanced/crawling/...
This was common even back in the mid-2000s:
https://www.avivadirectory.com/bethebot/
https://developers.google.com/search/blog/2006/09/how-to-ver...
https://developers.google.com/search/docs/advanced/crawling/...
As inflammatory as the headline of the page looks, they literally admit it's not google's fault in the smaller text lower down:
"This isn’t illegal and it isn’t Google’s fault, but"
Googlebot has a range of IP addresses that it publicly announces so websites can whitelist them.
Google says[1] they do not do this:
"Google doesn't post a public list of IP addresses for website owners to allowlist."
[1]https://developers.google.com/search/docs/advanced/crawling/...
If I actually wanted to restrict bots, it would be much easier to restrict googlebot because they actually follow the rules.
I don't disagree in principle that there should be an open index of the web, but for once I don't see Google as a bad actor here.