News for hackers. Stuff that matters.
Re:Cloudflare Radar by eminence32 on Wednesday September 30, 2020 @10:28AM (#24639048)
There's a lot of data here to digest, but one thing that stood out to me is:

> Bot traffic: 41% of total

So of all the traffic that CF sees, it attributes 41% of it to automated bots. This is way higher than I would have guessed!

Re:Cloudflare Radar by yjftsjthsd-h on Wednesday September 30, 2020 @12:14PM (#24640335)
> So of all the traffic that CF sees, it attributes 41% of it to automated bots. This is way higher than I would have guessed!

Bear in mind that that's what CF attributes to bots. As a human who gets flagged as a bot sometimes, I'm a little skeptical of their numbers.

Re:Cloudflare Radar by maxk42 on Wednesday September 30, 2020 @12:59PM (#24640911)
Around ten years ago I built the analytics system for an Alexa 100 company. The company had a policy of supporting IE6 until its share of traffic was below 1% and it was sitting at around 4% at the time IIRC. This was the figure that was reported industry-wide and this was also what we were seeing internally in our stats. However, partway through the project we decided to switch from an image pixel to javascript tracking. At that point we saw the share of IE6 traffic drop immediately to below 1% and it dawned on me what was happening: A lot of bot creators had been using an IE6 user agent to disguise their activity. This was in the days before headless browsing so the vast majority of them weren't downloading / processing javascript. I thought 4% was high then for bot activity but it wouldn't surprise me if the share has exploded since then.
Re:Cloudflare Radar by yjftsjthsd-h on Wednesday September 30, 2020 @02:38PM (#24642077)
Oh, I don't doubt that bot activity is rampant. The question is what Cloudflare's false-positive and false-negative rates are; I know from personal experience that they sometimes are convinced that humans are bots, and I assume sometimes bots get by them and therefore get counted as humans. But between those, my personal conclusion is that CF's claims about bot activity should be read with error bars.
Re:Cloudflare Radar by eli on Wednesday September 30, 2020 @02:59PM (#24642353)
Yup. Now imagine how many bot frameworks are smart enough to keep up to date with user agent strings.

Pro-tip in Google Analytics you can filter by some device color depth = 0 bits and find (what I assume) are all bots that way.

Re:Cloudflare Radar by vengefulduck on Wednesday September 30, 2020 @02:46PM (#24642168)
Also keep in mind it was probably a subset of bots using that user agent, so the total bot traffic could have been higher than 4%.
Re:Cloudflare Radar by eli on Wednesday September 30, 2020 @02:58PM (#24642344)
Sure and no doubt some bots pass as human.

But keep in mind CF has a lot of options for filtering traffic. Just because you got a CAPTCHA on a CF site doesn't necessarily mean considered you a bot.

Re:Cloudflare Radar by nuker on Thursday October 01, 2020 @03:42AM (#24648427)
> Just because you got a CAPTCHA on a CF site doesn't necessarily mean considered you a bot.

What else it thinks I am?

Re:Cloudflare Radar by chomp on Wednesday September 30, 2020 @10:36AM (#24639127)
I used to work at a webhosting company, crawlers were an enormous amount of our traffic. Some of our users, their only traffic was crawler traffic.
Re:Cloudflare Radar by eminence32 on Wednesday September 30, 2020 @10:41AM (#24639177)
Were these crawlers for big well-known things like google/bing/archive.org? I'm wondering if there are other types of crawlers that I don't even know about.
Re:Cloudflare Radar by drcongo on Wednesday September 30, 2020 @11:11AM (#24639544)
A site for one of my clients was, at one point, getting absolutely hammered by a bot that crawls for plagiarised content. It was ignoring our robots.txt so I ended up blocking it via CloudFlare. There are bots for pretty much anything you can think of out there.

edit: replaced "scrapes" with "crawls" for accuracy.

Re:Cloudflare Radar by oefrha on Wednesday September 30, 2020 @11:01AM (#24639416)
Traders crawl for financial information. Businesses (e.g. airlines) crawl for competitor pricing. Etc. etc.
Re:Cloudflare Radar by thinkloop on Wednesday September 30, 2020 @10:41AM (#24639179)
Why do bots need to crawl so much? Any idea what kind of bots they are, is it mainly search engines?
Re:Cloudflare Radar by heimatau on Wednesday September 30, 2020 @12:29PM (#24640519)
> Why do bots need to crawl so much?

The Earth has a (mostly) free society. Therefore, many programmers find a web crawler to be an advantageous endeavor. From search engines, to social media companies, to ad networks, to government agencies, to financial institutions, to students learning about programming, to nefarious actors, and even shadow versions of everything above. Bots are neat. One can do a lot with a freely accessible public database of information, only limit is one's creativity.

(edited: for formatting)

Re:Cloudflare Radar by pilsetnieks on Wednesday September 30, 2020 @03:53PM (#24643009)
> only limit is one's creativity

On the consumer side. There is a hard limit on bandwidth, though, on the provider side that can be exhausted by an overwhelming amount of bot traffic. In effect, that's what a DDoS is, even if unintentional.

Re:Cloudflare Radar by blakesterz on Wednesday September 30, 2020 @10:38AM (#24639147)
That's actually lower that I would have guessed. And lower than what I see on my servers. It's more than 1/2 for sure, and it's hard to guess how much of that other (less than half) part is real people, and not just bots pretending to be people. My 1/2 educated guess is about 40% of traffic to the sites I run is real people on any given day. These are all smallish websites for non-profits, so maybe not an "average" website, whatever that might mean.
Re:Cloudflare Radar by sgc on Wednesday September 30, 2020 @10:57AM (#24639371)
Filter by your country. The US for example has 57% bot activity.
Re:Cloudflare Radar by blakesterz on Wednesday September 30, 2020 @01:39PM (#24641351)
Aha! That sounds about right. Interesting, I didn't notice you can filter!
Re:Cloudflare Radar by tiffanyh on Wednesday September 30, 2020 @12:16PM (#24640363)
How did you filter by country?
Re:Cloudflare Radar by sgc on Wednesday September 30, 2020 @12:56PM (#24640867)
There is a search box at top of center graph.
Re:Cloudflare Radar by CoffeeDregs on Wednesday September 30, 2020 @01:55PM (#24641538)
Hadn't noticed that but that's a good point. "Way higher": I would have guessed the same as you but, upon further thought, I guess I'm not surprised: creating a crawler is 100 lines of Python so I would guess that there are actually tens of thousands of bots running around grabbing web pages. Humans are an ever diminishing portion of the "web", especially given the (old) focus on "semantic web" and machine-friendly-ish formats...
Re:Cloudflare Radar by BuildTheRobots on Wednesday September 30, 2020 @10:39AM (#24639150)
Really interesting numbers. I just wish it was possible to see all the other statistics in relation to letgit vs bot traffic.

eg if mobile traffic is 35% of total, what percentage of legit vs bot traffic is mobile? TLS 1.2 vs 1.3 is a 38%/61% split, but how much bot traffic is encrypted?

Re:Cloudflare Radar by jgrahamc on Wednesday September 30, 2020 @10:40AM (#24639165)
This is just the beginning. We are going to be building out more and more functionality. My hope is that we deploy jupyter and let people crunch numbers in the ways they choose.
Re:Cloudflare Radar by cortesoft on Wednesday September 30, 2020 @03:49PM (#24642966)
A single bot can generate thousands of times more traffic than a person. Even a small number of bots could easily generate a huge percentage of traffic.
Re:Cloudflare Radar by tiffanyh on Wednesday September 30, 2020 @12:18PM (#24640382)
I wonder if there is a correlation between Cloudflare customers and internet properties who'd be targeted by bots.
Re:Cloudflare Radar by llacb47 on Wednesday September 30, 2020 @12:29PM (#24640527)
They probably flag a lot of Tor users as bots.