Re:Only Google is really allowed to crawl the web
by judge2020 on Friday March 26, 2021 @01:49PM
(#26594920)
> Googlebot's UA worked just fine however
They obviously don't care enough then - Google says you should use rdns to verify that googlebot crawls are real[0]. Cloudflare does this automatically now as well for customers with WAF (pro plan).
They obviously don't care enough then - Google says you should use rdns to verify that googlebot crawls are real[0]. Cloudflare does this automatically now as well for customers with WAF (pro plan).
0: https://developers.google.com/search/docs/advanced/crawling/...