News for hackers. Stuff that matters.
Re:We can't have nice things because of AI scrapers by bitbasher on Tuesday January 13, 2026 @05:48PM (#46609473)
Wow, so to prevent AI scrapers from harvesting my data I need to send all of my traffic through a third party company that gets to decide who gets to view my content. Great idea!
Re:We can't have nice things because of AI scrapers by Aurornis on Tuesday January 13, 2026 @06:16PM (#46609834)
You don’t need to do anything. You can use any number of solutions or roll your own.

Someone shared an alternative. Must everything in AI threads be so negative and condescending?

Re:We can't have nice things because of AI scrapers by thefz on Wednesday January 14, 2026 @02:12AM (#46613253)
> Must everything in AI threads be so negative and condescending?

Because if I own a website or a service and it is being degraded or slowed by some third party tool that wants to slurp its content for his own profit and don't even share, I tend to be irritated. And AI apologists/evangelists don't help when they try to justify the behavior.

Re:We can't have nice things because of AI scrapers by dannersy on Wednesday January 14, 2026 @02:28AM (#46613331)
Yes, they could roll their own, but you have no issues with this being necessary? I think the attitude of "just deal with it" is far more negative than someone expressing they are upset with the state of the internet, its controllers, and its abusers.
Re:We can't have nice things because of AI scrapers by TitaRusell on Wednesday January 14, 2026 @05:15AM (#46614337)
There's trillions invested in AI. Don't expect any introspective insight or criticism about it.
Re:We can't have nice things because of AI scrapers by bs7280 on Wednesday January 14, 2026 @04:57PM (#46624251)
This is like saying "lets just get rid of all the guns" to solve gun violence and gun crime in the USA. The cat is out of the bag and no one can put it back. We live in a different world now and we have to figure it out.
Re:We can't have nice things because of AI scrapers by rester324 on Tuesday January 13, 2026 @05:56PM (#46609576)
You can implement this yourself, who is stopping you?
Re:We can't have nice things because of AI scrapers by zzzeek on Tuesday January 13, 2026 @06:01PM (#46609637)
Citation needed
Re:We can't have nice things because of AI scrapers by zimpenfish on Tuesday January 13, 2026 @06:10PM (#46609758)
I use iocaine[0] to generate a tarpit. Yesterday it served ~278k "pages" consisting of ~500MB of gibberish (and that's despite banning most AI scrapers in robots.txt.)

[0] https://iocaine.madhouse-project.org

Re:We can't have nice things because of AI scrapers by chao- on Tuesday January 13, 2026 @06:41PM (#46610137)
Can't seem to access this.

It flashes some text briefly then gives me an 418 TEAPOT response. I wonder if it's because I'm on Linux?

EDIT: Begrudgingly checked Chrome, and it loads. I guess it doesn't like Firefox?

Re:We can't have nice things because of AI scrapers by dpkirchner on Tuesday January 13, 2026 @07:48PM (#46610847)
Nor Safari on iOS.
Re:We can't have nice things because of AI scrapers by zimpenfish on Wednesday January 14, 2026 @03:46AM (#46613793)
Works fine on my iOS Safari - maybe there's some extension that's tickling it just the wrong way?
Re:We can't have nice things because of AI scrapers by dpkirchner on Wednesday January 14, 2026 @08:14AM (#46615682)
It still fails with all of my extensions disabled (wipr, privacy redirect). I just get a download dialog. I don't know what the HTTP status code is, however.

I found a flagged HN submission about it and it has just about the same result for me and for others. My first tap failed in a weird way (showed some text then redirected quickly to its git repo) and all subsequent taps trigger a download.

https://news.ycombinator.com/item?id=44538010

Given the replies here and in the older post, I suspect this tool is doing more harm to humans than to bots.

Re:We can't have nice things because of AI scrapers by zephen on Tuesday January 13, 2026 @07:23PM (#46610633)
Doesn't work on my firefox either.

Friendly fire, I suppose.

Re:We can't have nice things because of AI scrapers by godelski on Tuesday January 13, 2026 @07:40PM (#46610778)
Works on my Firefox. Mac and Linux
Re:We can't have nice things because of AI scrapers by doublerabbit on Tuesday January 13, 2026 @06:40PM (#46610133)
Unfortunately and you kind of have to count this as the cost of the Internet. You've wasted 500Mb of bandwidth.

I've had colocation for eight years+. My monthly b/w cost is now around 20-30Gb a month given to scrapers where I was only be using 1-2Gb a month, years prior.

I pay for premium bandwidth (it's a thing) and only get 2TB of usable data. Do I go offline or let it continue?

Re:We can't have nice things because of AI scrapers by zimpenfish on Wednesday January 14, 2026 @04:09AM (#46613937)
> You've wasted 500Mb of bandwidth.

Yep, it sucks, but on the positive side, I'm feeding 500Mb of garbage into them every day and that feels like enough of a small win for me.

> My monthly b/w cost is now around 20-30Gb a month given to scrapers [...] 1-2Gb a month

That definitely sucks.

> Do I go offline or let it continue?

Might be time to start blocking entire IP ranges and ASNs and see if that helps.

Re:We can't have nice things because of AI scrapers by zzzeek on Wednesday January 14, 2026 @12:22PM (#46618946)
i have no idea what this does because the site is rejecting my ordinary firefox browser with "Error code: 418 I'm a teapot". Even from a private browser.

If I hit it with Chrome, now I can see a site.

Seems pretty not ready for prime time as a lot of my viewers use Firefox

Re:We can't have nice things because of AI scrapers by godelski on Tuesday January 13, 2026 @07:36PM (#46610736)
One of the most popular ones is Anubis. It uses a proof of work and can even do poisoning: https://anubis.techaro.lol/

They even mention iocaine. I know, inconceivable!: https://iocaine.madhouse-project.org/

There's also tons of HN posts on the topic with varying solutions:

https://news.ycombinator.com/item?id=45935729

https://news.ycombinator.com/item?id=45711094

https://news.ycombinator.com/item?id=44142761

https://news.ycombinator.com/item?id=44378127

Re:We can't have nice things because of AI scrapers by zzzeek on Tuesday January 13, 2026 @10:05PM (#46611792)
Anubis is the only tool that claims to have heuristics to identify a bot, but my understanding is that it does this by presenting obnoxious challenges to all users. Not really feasible. Old school approaches like ip blocking or even ASN blocking are obsolete - these crawlers purposely spam from thousands of IPs, and if you block them on a common ASN, they come back a few days later from thousands of unique ASNs. So this is not really a "roll your own" situation, especially if you are running off the shelf software that doesn't have some straightforward means of building in these various approaches of endless page mazes (which I would still have to serve anyway).
Re:We can't have nice things because of AI scrapers by GuinansEyebrows on Tuesday January 13, 2026 @06:09PM (#46609747)
https://forge.hackers.town/hackers.town/nepenthes

> Citation needed

this reply kinda sucks :)