News for hackers. Stuff that matters.
Re:I was wrong about robots.txt by komali2 on Thursday July 17, 2025 @03:36AM (#44590716)
I'm confused why scraping is so resource intensive - it hits every URL your site serves? For an individual ecommerce site that's maybe 10,000 hits?
Re:I was wrong about robots.txt by xnorswap on Thursday July 17, 2025 @04:06AM (#44590894)
And the thousands of other bots also hitting those, together is far more than the legitimate traffic for many sites.
Re:I was wrong about robots.txt by TylerE on Thursday July 17, 2025 @05:00AM (#44591225)
Yeah, there were times, even running a fairly busy site, that the bots would outnumber user traffic 10:1 or more, and the bots loved to endlessly troll through things like archive indexs that could be computationally (db) expensive. At one point it got so bad that I got permission to just blackhole all of .cn and .ru, since of course none of those bots even thought of half obeying robots.txt. That literally cut CPU load on the database server by more than half.
Re:I was wrong about robots.txt by EPendragon on Thursday July 17, 2025 @09:38AM (#44593264)
For me (as I'm sure for plenty other people as well) limiting traffic to actual users matters a lot because I'm using a free tier for hosting in the time being. Bots could quickly exhaust it, and your website could be unavailable for the rest of the current "free billing" cycle, i.e. until your quota gets renewed.
Re:I was wrong about robots.txt by hombre_fatal on Thursday July 17, 2025 @10:48AM (#44594030)
In the last month, bot traffic has exploded to 10:1 due to LLM bots on my forum according to Cloudflare.

It would be one thing if it were driving more users to my forum. But human usage hasn't changed much, and the bots drop cache hit rate from 70% to 4% because they go so deep into old forum content.

I'd be curious to see a breakdown of what the bots are doing. On demand searches? General data scraping? I ended up blocking them with CF's Bot Blocker toggle, but I'd allow them if it were doing something beneficial for me.

Re:I was wrong about robots.txt by paulddraper on Thursday July 17, 2025 @10:51AM (#44594059)
Yes. That’s a lot of bandwidth, depending on the content of course.
Re:I was wrong about robots.txt by jlokier on Thursday July 17, 2025 @01:04PM (#44595525)
With 1000s of bots per month and 10,000 hits on an ecommerce site, with product images, that's a lot of data transfer, and a lot of compute if your site has badly designed or no caching, rendering all the same page components millions of times over. But...

Part of the problem is all those companies who use AWS "standard practice" services, who assume the cost of bandwidth is just what AWS charges, and compute-per-page is just what it is, and don't even optimise those (e.g. S3/EC2/Lambda instead of CloudFront).

I've just compared AWS egress charge against the best I can trivially get at Hetzner (small cloud VMs for bulk serving https cache).

You get an astonishing 392x(!) more HTTPS egress from Hetzner for the same price, or equivalently 392x cheaper for the same amount.

You can comfortably serve 100+ TB/month that way. With 10,000 pages times 1000 bots per month, that gives you 10MB per page, which is more than almost any eCommerce site uses, when you factor that bots (other than very badly coded bots) won't fetch the common resources (JS etc.) repeatedly for each page, only the unique elements (e.g. HTML and per-product images).

Re:I was wrong about robots.txt by hinkley on Thursday July 17, 2025 @01:24PM (#44595753)
You’ve forgotten the combinatorics of query params.

10,000 is Monday. Morning.