A sad state of affairs, but it was predicted decades ago that commercial interests would turn the internet into what it is today, even without AI. Layer on the dead internet theory slowly coming true, and walled gardens almost feel like the last bastion of free internet, rather than being what brought it to an end.
There's probably some nuance there, maybe the walled gardens allowed us to be comfortable letting it get this bad. Either way, what's gone is gone.
I am getting a lot of joy from local net and just making little devices at home, that is giving me the same excitement that the web did in the past!
Started my career as a web developer and always have a soft spot for it but from a hobby developer standpoint hosting and deploying a site feels like an uphill battle with little upside.
Theres a lot to not like about app development but after you get approval for your app it's pretty hands off.
Heck on android you can just distribute the raw APK on a Google drive. Or just keep the app locally for yourself.
I probably have about 20 self made apps on my phone and they each give me a bit of happiness.
For about a year, after which point both Apple and Google will arbitrarily remove your app from stores for not keeping up with the platform update treadmill.
Keep up with platform updates or constantly deal with the afformentioned crawlers and hugs of death.
Would something like Cloudflare help with bot detection?
Part of the problem is the economics of it -- I've chosen to self-fund a high traffic site without ads, and that's on me. But it was possible to do this just a few years ago.
Cloudflare no longer does CAPTCHAs so even if users get flagged as bots, the user experience isn't terrible. You just have to click on a box and you're on your way. It adds maybe 3s of delay, far better than anti-bot solutions that require you to solve an captcha, or imperva's (?) challenge that requires you to hold a button for 5-10s seconds.
You'll be flagged as a bot if your browser configuration has something "weird" (e.g. webrtc is disabled to reduce your attack surface) and you will be completely unable to access any site behind cloudflare with the anti-bot options turned on. You'll get an infinite redirect loop, not a button to click.
The researcher who discovered this was able to generate 60,000 "I am not a bot" cookies per day, and use them up about 15 times each in a bot before it started getting captchas.
That was in 2016 though.
(My site is basically a search engine, which complicates matters because there's effectively an infinite space of URLs. Just one of these rogue bots can scrape millions of pages from tens of thousands of IPs; and I think there are hundreds of the bots at any given moment...)
I like the Cloudflare challenge ideas suggested on this thread, though, I might try them again.
On my personal site I started blocking bots and also set a "noindex, nofollow" rule to block web crawlers and search bots too. I noticed no change in traffic and still do about 10,000 visits a month.