Re:Cloudflare builds AI to lead AI scraper bots into horribl...
by czk on Saturday March 22, 2025 @10:54AM
(#43446092)
Yeah you are right about the caching thing, I understand what CF is doing I just cant help but feel like there is something deeply wrong about this approach. But I don't have any better ideas to propose. Maybe I'm just holding on too tight to the 'old web' that I've known and loved.
In the previous discussions, one of the big AI problems was that they follow every link [0], including expensive ones like "git blame" view.. as well as "every commit of every log", which can be easily in millions if you have mirrors of few repos with many branches. It's simply impossible to cache this all at CloudFlare, and solution like AI maze (or outright blocking) seem like the only way.
I feel you about "requests one-liners in python" - I am still running a few of those in crontab on some server.. but their time might be past. The "average guy" does not care about this kind of scripting, so we will get lumped together with evil AI and blocked.
[0] https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...