Re:Cloudflare Radar
by chomp on Wednesday September 30, 2020 @10:36AM
(#24639127)
I used to work at a webhosting company, crawlers were an enormous amount of our traffic. Some of our users, their only traffic was crawler traffic.
[ Parent
| on HN ]
Re:Cloudflare Radar
by eminence32 on Wednesday September 30, 2020 @10:41AM
(#24639177)
Were these crawlers for big well-known things like google/bing/archive.org? I'm wondering if there are other types of crawlers that I don't even know about.
A site for one of my clients was, at one point, getting absolutely hammered by a bot that crawls for plagiarised content. It was ignoring our robots.txt so I ended up blocking it via CloudFlare. There are bots for pretty much anything you can think of out there.
edit: replaced "scrapes" with "crawls" for accuracy.
Traders crawl for financial information. Businesses (e.g. airlines) crawl for competitor pricing. Etc. etc.
Why do bots need to crawl so much? Any idea what kind of bots they are, is it mainly search engines?
> Why do bots need to crawl so much?
The Earth has a (mostly) free society. Therefore, many programmers find a web crawler to be an advantageous endeavor. From search engines, to social media companies, to ad networks, to government agencies, to financial institutions, to students learning about programming, to nefarious actors, and even shadow versions of everything above. Bots are neat. One can do a lot with a freely accessible public database of information, only limit is one's creativity.
(edited: for formatting)
Re:Cloudflare Radar
by pilsetnieks on Wednesday September 30, 2020 @03:53PM
(#24643009)
> only limit is one's creativity
On the consumer side. There is a hard limit on bandwidth, though, on the provider side that can be exhausted by an overwhelming amount of bot traffic. In effect, that's what a DDoS is, even if unintentional.