Re:Cloudflare Radar
by eminence32 on Wednesday September 30, 2020 @10:41AM
(#24639177)
Were these crawlers for big well-known things like google/bing/archive.org? I'm wondering if there are other types of crawlers that I don't even know about.
[ Parent
| on HN ]
A site for one of my clients was, at one point, getting absolutely hammered by a bot that crawls for plagiarised content. It was ignoring our robots.txt so I ended up blocking it via CloudFlare. There are bots for pretty much anything you can think of out there.
edit: replaced "scrapes" with "crawls" for accuracy.