News for hackers. Stuff that matters.
Re:Messing with Scraper Bots by NoiseBert69 on Saturday November 15, 2025 @04:33AM (#45936210)
Hm.. why not using dumbed down small, self-hosted LLM networks to feet the big scrapers with bullshit?

I'd sacrifice two CPU cores for this just to make their life awful.

Re:Messing with Scraper Bots by Findecanor on Saturday November 15, 2025 @06:55AM (#45936810)
You don't need an LLM for that. There is a link in the article to an approach using Markov chains created from real-world books, but then you'd let the scrapers' LLMs re-enforce their training on those books and not on random garbage.

You could make a list of words from each word class, and a list of sentence structures where each item is a word class. Pick a pseudo-random sentence; for each word class in the sentence, pick a pseudo-random word; output; repeat.

Be sure to add delays to your server so that you don't induce more demand. Traffic isn't free.

Re:Messing with Scraper Bots by qezz on Saturday November 15, 2025 @06:38AM (#45936737)
That's very expensive.
Re:Messing with Scraper Bots by mnau on Saturday November 15, 2025 @07:35PM (#45941755)
He addresses that. Basically, there are gatekeepers and if you get on the wrong side of them, only manual intervention can save you. And we all know how Google loves providing a human to resolve problems.

> I came to the conclusion that running this can be risky for your website. The main risk is that despite correctly using robots.txt, nofollow, and noindex rules, there's still a chance that Googlebot or other search engines scrapers will scrape the wrong endpoint and determine you're spamming.