News for hackers. Stuff that matters.
Re:I rebuilt my blog's cache. Bots are the audience now by cullumsmith on Sunday May 03, 2026 @11:49AM (#47998155)
I simply block all AI crawlers with a user-agent check in nginx.conf.
Re:I rebuilt my blog's cache. Bots are the audience now by microtonal on Sunday May 03, 2026 @12:03PM (#47998313)
I also block all AI crawlers. I am not sure why I should give them my content for them to rip it off and make money from it through training or agents. Sadly, a lot of AI companies are trying to make requests indistinguishable from regular browsers from residential connections, so unfortunately I have to use Cloudflare to block them.

Ideally I'd make the content available to crawlers for training open models, but that seems to be nearly impossible. It would be possible if other AI companies behaved.

Re:I rebuilt my blog's cache. Bots are the audience now by Barbing on Sunday May 03, 2026 @12:18PM (#47998481)
>so unfortunately I have to use Cloudflare to block them.

That can’t block Grok, can it?

(You might have a fake iPhone or something visit your site if you ask Grok to retrieve information from it)

Re:I rebuilt my blog's cache. Bots are the audience now by tardedmeme on Sunday May 03, 2026 @01:31PM (#47999323)
What's the IP address of the supposed iPhone? Does it come from T-Mobile or from xAI?
Re:I rebuilt my blog's cache. Bots are the audience now by Barbing on Sunday May 03, 2026 @02:01PM (#47999613)
Residential I thought? It might’ve been even someone on here who posted about watching their server logs while they messaged Grok themselves.

Curious if xAI has a phone farm. Maybe just running simulators on servers?

Re:I rebuilt my blog's cache. Bots are the audience now by tardedmeme on Monday May 04, 2026 @01:31PM (#48011934)
Residential proxies are a commodity at this point. You can also run your own network and try to get it misclassified as residential.
Re:I rebuilt my blog's cache. Bots are the audience now by orf on Sunday May 03, 2026 @12:08PM (#47998362)
*some AI crawlers. Not many
Re:I rebuilt my blog's cache. Bots are the audience now by robhoeijmakers on Sunday May 03, 2026 @11:58AM (#47998258)
I started blocking some of them. But for now I want to improve visibility before further blocking or optimising. The dashboard helps with this.
Re:I rebuilt my blog's cache. Bots are the audience now by tardedmeme on Sunday May 03, 2026 @01:31PM (#47999307)
This works for a few weeks to months. Then they detect your site is hostile to them and enable evasion mode, with random IP addresses and user-agent strings. Proxies are expensive so at least they're losing money.