News for hackers. Stuff that matters.
Re:Google Search Is Dying. What Comes Next Is Worse by neuralkoi on Monday August 10, 2026 @09:11PM (#49252082)
I've seen websites put up some draconian measures to try and get a grip on the scraping. So much for the sub-second loading experience when you have Cloudflare, Google, Anubis, and all these other captcha services trying to see if you're a human. It's made the web browsing experience so much worse.

Some of the proposals to address this include charging bots for access to web resources, but they will also have repercussions for regular users. I don't see how you solve this cleanly.

Re:Google Search Is Dying. What Comes Next Is Worse by JKCalhoun on Monday August 10, 2026 @09:32PM (#49252262)
Me, I'm just scraping the parts of the internet I like, toying with local LLMs… ready really to just shove off.
Re:Google Search Is Dying. What Comes Next Is Worse by Grimburger on Tuesday August 11, 2026 @02:50AM (#49254274)
Cloudflare specifically has a block for LLM and AI training bots now.

Not sure of the effectiveness but it's there.

Re:Google Search Is Dying. What Comes Next Is Worse by matherial on Tuesday August 11, 2026 @03:07AM (#49254406)
Minimal. I'm behind Cloudflare and 90% of the traffic is still scrapers. I don't think they're serious about the long tail.

I think the main thing Cloudflare is trying to do is block direct traffic from frontier labs and then start charging them for access. They might end up shooting themselves in the foot, as this simply empowers sketchy residential-proxy outfits to undercut Cloudflare and sell the data to labs for less.

Re:Google Search Is Dying. What Comes Next Is Worse by inigyou on Tuesday August 11, 2026 @06:59AM (#49256226)
I think the other thing they're trying to do is get most of the internet to send them all of their cleartext traffic. Expect in 2040 the PRISM2 docs will get leaked by some Eduardo Rainedon and we'll find out Cloudflare was the NSA all along.
Re:Google Search Is Dying. What Comes Next Is Worse by bakugo on Tuesday August 11, 2026 @04:19AM (#49254897)
It still only blocks "well-behaved" bots that have proper User-Agents and respect robots.txt, so it's largely pointless.

The problematic bots are all disguising themselves as Chrome and sending requests from millions of residential proxy IPs, and the only real solution to those is some sort of captcha or PoW page on first visit.

Re:Google Search Is Dying. What Comes Next Is Worse by satvikpendem on Tuesday August 11, 2026 @10:21AM (#49258882)
While they themselves announced an AI bot, the irony is palpable. In reality they just want to control who gets access to what, themselves excepted.
Re:Google Search Is Dying. What Comes Next Is Worse by robinsonb5 on Tuesday August 11, 2026 @03:22AM (#49254509)
Sure - it sucks, unfortunately the alternative is the sites going away entirely. When the load from scraper bots is constantly knocking the site offline the choices are literally to allow it to remain inaccessible for much of the time, put up a layer of defenses with all the user-annoyance compromises that entails, or just give up and unpublish the site.
Re:Google Search Is Dying. What Comes Next Is Worse by Borg3 on Tuesday August 11, 2026 @03:28AM (#49254558)
The alternative is simple.. Go dark. VPN tech is known from like 30 years. Pretty much everyone can use it (VPN providers). But instead using it to browse net, build VPN overlay networks of interest for people. Gaming networks, R&D networks, Retro Networks. People will peer to PoP and use resources. Bad actor? BAN it from network. You have control. This could be done in Internet, but big corpos and big money won the battle. Just wake F*ing up...
Re:Google Search Is Dying. What Comes Next Is Worse by kukkeliskuu on Tuesday August 11, 2026 @04:54AM (#49255124)
Continuing on your suggestion.

There could be open source tooling to create custom private "closednets", with

- trust ring mechanism to allow invitations, flagging, banning, and banning those that invite people who were banned

- the rules of the closednet

- search engine with opt-in scraping

- portal (remember the 80s?) with all the registered nodes, perhaps by service category such as public git repo hosts, web sites etc.

etc.

The first closednet could be Hacker News.

Re:Google Search Is Dying. What Comes Next Is Worse by inigyou on Tuesday August 11, 2026 @06:58AM (#49256211)
It doesn't have to be an IP-layer network. A website that you need to log in to view works just as well.
Re:Google Search Is Dying. What Comes Next Is Worse by Borg3 on Tuesday August 11, 2026 @08:35AM (#49257303)
Nah, it needs to be IP. IP is well estabilished protocol, everything speak it. Once you set it up, you can use it whatever you like. Web pages, gaming service, IRC, Mail, P2P confereces, everything. Everyone will bring it own slice to the pie. You love networking, became PoP and peer and provide access. You just want content? Connect to closest PoP, get IP + DNSproxy and vioala.
Re:Google Search Is Dying. What Comes Next Is Worse by inigyou on Tuesday August 11, 2026 @08:40AM (#49257363)
The networking part exists, it's called DN42. But it's hard to set up. I don't see why you'd use this instead of an application layer password.
Re:Google Search Is Dying. What Comes Next Is Worse by Borg3 on Tuesday August 11, 2026 @08:50AM (#49257481)
I know DN42, but this network is more oriented toward R&D and experimenting. Yeah, its not for your avarage Joe. But your avarage Joe can buy connection from VPN provider and use it, and so we can provide user friendly PoPs with minimal skills needed to setup, supporting different VPN software.
Re:Google Search Is Dying. What Comes Next Is Worse by inigyou on Tuesday August 11, 2026 @11:01PM (#49267366)
or you put a password on a website
Re:Google Search Is Dying. What Comes Next Is Worse by Borg3 on Wednesday August 12, 2026 @04:08AM (#49269214)
Spoken like a true websurfer. You know, networking is not only about WWW.
Re:Google Search Is Dying. What Comes Next Is Worse by inigyou on Wednesday August 12, 2026 @07:43AM (#49270875)
You can also put passwords on things that aren't websites.
Re:Google Search Is Dying. What Comes Next Is Worse by iamcalledrob on Tuesday August 11, 2026 @07:39AM (#49256627)
I think you'd struggle to keep LLM bots off the network unfortunately.

If it had any real value, anyway.

Small, truly private communities could be an interesting thing though.

Re:Google Search Is Dying. What Comes Next Is Worse by Borg3 on Tuesday August 11, 2026 @08:32AM (#49257275)
How I can strugle to keep them off? To peer to network, you need to talk to human. Arrange L2 connection, assign IPs (only static). If its leaf node, we are done. If its another network, we need to form BGP connections to exchange routing.

Yeah, Network by Humans for Humans. Thats why Im not interested in all those IoT/Auto networks when you just connect and stuff automagically configure. It looks nice at first glance, but you loose control. F2F works way better in that matter, like RetroShare, but I never investigated it much.

Re:Google Search Is Dying. What Comes Next Is Worse by gosub100 on Tuesday August 11, 2026 @08:30AM (#49257245)
I just used a VPN yesterday and the NY times blocked me because they think I look like a bot. It gave a couple possible reasons, one being "a bot was also using this IP address".

VPNs are great for torrenting but any serious website like an online bank or web email provider will turn you away. They claim it's for bots but really it because they only want customers they can track.

Re:Google Search Is Dying. What Comes Next Is Worse by inigyou on Wednesday August 12, 2026 @07:44AM (#49270882)
You're conflating a VPN as in a virtual private network, with a VPN as in Nord/Atlas/ExpressVPN, which use the same underlying technology in a different way.
Re:Google Search Is Dying. What Comes Next Is Worse by kees99 on Tuesday August 11, 2026 @05:52AM (#49255630)
> Cloudflare, Google, Anubis, and all these other captcha services trying to see if you're a human.

Yep. IMO, this is so far the biggest AI-inflicted damage to the web. A bit of anecdata - wikipedia (and all other wikimedia sites) are blocking my Firefox since about a week, with a "please respect our bot policy" message. Outright block, not even a captcha.

It took me a while to figure out they don't like me disabling some SSL ciphers, so now "JA4 browser fingerprint" is not matching user-agent. Funnily enough curl (what I would imagine a bot would use) pulls exact same URLs from exact same client IP, just fine.

Re:Google Search Is Dying. What Comes Next Is Worse by ImaCake on Tuesday August 11, 2026 @07:39AM (#49256623)
Maybe the curl thing is because they are happy to let you do some light scraping. What they want to avoid is bots directly crawling the page interactively. No one seems to be blocking chatGPT when I promot it to use it's web search skill anyway.