News for hackers. Stuff that matters.

A comment on An Update on the scraper situation. Parent comment

Re:An Update on the scraper situation by corbet on Friday July 10, 2026 @05:19PM (#48865417)
Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.
[ Parent | on HN ]
Re:An Update on the scraper situation by everfrustrated on Friday July 10, 2026 @05:31PM (#48865581)
Your earlier article suggests you aren't using a CDN. Might be well worth looking into - not for any bot detection so much as just having a good old fashioned cache in front of you.
[ Parent | on HN ]
Re:An Update on the scraper situation by mplewis on Saturday July 11, 2026 @01:15AM (#48868974)
As someone who operates a wiki, this does not solve the problem.
[ Parent | on HN ]
Re:An Update on the scraper situation by solid_fuel on Saturday July 11, 2026 @07:19AM (#48870996)
Caches only help for pages that have been requested recently. The behavior of crawlers - going from one page to the next across the whole site - will probably not be mitigated significantly by a cache.
[ Parent | on HN ]