I really wouldn't be surprised to learn that 46% was just Bing's crawler...

Considering the bulk of data is now behind some form of paywall (ie you need a netflix subscription to access netflix data) I would be very surprised to see even bing+google crawlers combined consuming anywhere near 50% of traffic. They just don't have the access necessary to start pulling such numbers.

I don’t quite understand how or why but I was tangentially involved in a small-mid size e-commerce website that only serves B2B customers in the US and to a small extent in Canada and it absolutely got lots of hits from what I assume are Chinese search engines to the point where we decided we absolutely needed cloud flare rate limiting.

Now the code is a hot mess, sure and I am partially to blame for that but that is kind of besides the point. We don’t need to serve any more than thousands of concurrent users which the website can handle but we have to basically ban Chinese traffic to stay online.

Maybe people at bigger companies already know this but it was a revelation to me how much it takes just to stay alive in production.

I anal and I definitely don’t know what they get out of crawling every single product detail page on our website multiple times a day. Nothing here changes that often. Maybe they have some bad/overzealous code? Are they looking to attack take over our servers to them attack others with our machines? If it is an attack, why use Chinese IP addresses? Why not use their bit farms? If it is legitimate search engine, why not respect robots.txt?

"If it is an attack, why use Chinese IP addresses?"

Because there is nothing, you can do about it anyway?

Even if you could proof, it is an attack, do you really would consider sueing some chinese IP adresses?

But I rather suspect, it is just bad crawler code.

I’d put my money on building early Dall-e and LLM training sets, followed by DDOS-for-hire.

I thought it would be more. A human doesn't visit anywhere near 10 different webpages a second, a bot can. A human isn't uploading and downloading data 24/7, a bot is. A human doesn't make a new post a minute, every minute, a bot does.

True but bots don't regularly stream hd videos from netflix/youtube.

Maybe to download and pirate the videos

Surely each group would only download a given video once though, compared to the millions of times they could be downloaded by legitimate users.

A bot can saturate text-based bandwidth as fast as a person saturates multimedia-based bandwidth.

I have speculated for a while that bots “watch” videos on YouTube to appear human so google doesn't ban their accounts.

Every heard of click farms ?

I tend to agree unless everyone else is using 32GB favicons on the default websites like me. Streaming takes up a massive amount of bandwidth. That and P2P torrent sharing is only increasing with all the streaming services going back to the cable TV payment plan models.

There are a handful of bots that mirror/archive multimedia content that is anonymously accessible. There is no way those bots have the storage capacity to mirror even a single pass of all the anonymous content.

What I have seen increasing exponentially is port scanning but that takes up almost no bandwidth. Even the broken scanners that in effect look like an amateur DDoS only utilize about 15kb/s using dozens of CIDR blocks at the same time. That does not even remotely hold a candle to streaming.

The link below [1] is talking about Netflix as a percentage of internet downstream traffic and this is only Netflix. There are now hundreds of streaming providers and according to Sandvine streaming accounts for 65% of internet traffic. [2] This does not include torrents and other file sharing.

Here [3] are some fun stats. One of them backs up OP.

Every day, the internet generates more than 2,183,908 tons of CO2 emissions.

Internet traffic statistics show that 51.8% of all traffic is generated by bots, while humans account for only 48.2%.

[1] - https://www.makeuseof.com/tag/how-much-of-the-internets-band...

[2] - https://www.tubefilter.com/2023/01/20/sandvine-video-data-ba...

[3] - https://dataprot.net/statistics/internet-statistics/

I once wrote a small utility for my team that ended up being the second or third largest API consumer in the company until someone yelled about it and I throttled back the polling significantly. I always think of The Sorcerer's Apprentice sequence from Fantasia - while we sleep, our automatons labor, potentially unceasingly.

And I can certainly imagine bots fetching Netflix or Porn or whatever video for personal archival purposes (I use youtube-dl to protect a few videos I really like from the vicissitudes of Google myself).

I think there are bots to watch youtube and twitch to boost the viewership. I think that could consume a lot of traffic. I have no data or research to back that up and I am no expert.

I have no idea either, but most mobile/residential proxy providers heavily charge bandwidth [1]. I'd imagine that you can bot views on twitch without streaming the video, since the iOS twitch app offers audio-/chat-only modes, so just connecting to the chat (+ mb some other obfuscated stuff) could be sufficient to bot views.

[1] $9.45/GB at brightdata.com/proxy-types/residential-proxies

With you. In 2021 53% of traffic was from 3 video platform - not counting countless other video providers and also not counting all other non-video content.

Our data show in the first half of 2021 bandwidth traffic was dominated by streaming video, accounting for 53.72% of overall traffic, with YouTube, Netflix, and Facebook video in the top three.

https://www.sandvine.com/hubfs/Sandvine_Redesign_2019/Downlo...

And looks like Sandvine is very credible as countries relies upon it to censor internet in their countries.

American Technology Is Used to Censor the Web From Algeria to Uzbekistan

https://www.bloomberg.com/news/articles/2020-10-08/sandvine-...

> And looks like Sandvine is very credible as countries relies upon it to censor internet in their countries.

Sandvine's primary business is traffic shaping and bandwidth management for cellular networks, shipboard networks, and other places where you need to do intelligent QoS. They are the reason Google Maps still works well on your cellular connection while you pass a house using cellular broadband to download a torrent.

The fact that some people use it to censor is an abuse of technology, not the intention of it.

Google for the UN Economist Network Report on the Attention Economy. There is a line there that says less than 1% of all content produced is actually consumed by ppl. And its shrinking.

When cost to broadcast for anyone on the net falls to 0 everything turns to shit.

It's true because advertisers and publishers measure "performance" (hence revenue/prices) based mostly on views & clicks. You will be amazed how many farms are in the open but hidden. And we're not talking on those stupid easy to catch traffic boosters, it's an entire industry behind.

Exactly. No way it's half of bandwidth, because video.

I'm guessing it's half of "connections", including DDoS attacks. But even then, I wonder how reliable their methodology is. Like are they including port scanning here, when a connection isn't even made?

I’d believe half of all traffic to non-video hosts, however. We provide a lot of free content and it’s staggering how many bad robots there are - not actual malice, just what you’d expect from lousy programmers who don’t get the bill. Things like the same IP / User-Agent downloading the same file thousands of times just in case it changed in the last 5 seconds or crawling millions of permutations of search parameters rather than using the site maps. Many week that’s half of the total traffic, many tens of terabytes of HTML & JSON.

As someone who once wrote a scraper to interface with a site that lists lawyers (So we can blacklist them in advance from being called, because they REALLY don't like getting contacted by accident), and another time a scraper to get a list of unemployed people (employee leasing): Bad bots really ruin it for those that actually try to make well behaved scrapers.

Yes. Stuff like the default Python user agent being blocked by Cloudflare cause hassles for students & are entirely due to sloppy bot developers.

The other possibility here then is that legitimate web page views pale in comparison to both streaming and bot traffic, and bot traffic is orders of magnitude higher than other web activity excluding streaming.

Boys need to be entertained as well. Bots have been such an issue Netflix started cracking down on password / account sharing between bots.