Why? What is the goal of a scraper, and how does disabling the source of the data benefit them?

> Why? What is the goal of a scraper, and how does disabling the source of the data benefit them?

The next scraper doesn’t get the data. People don’t realize we’re not compute limited for ai, we’re data limited. What we’re watching is the “data war”.

at this point we're _good data_ limited, which has little to do with scraping.

Scraping social media is good data, even without ML. The fact that something is "happening" to people in a social space inherently has importance to people. The specter of law is more threatening to whether companies can get their hands on good data.

Why kind of data that isn’t public would be so valuable for AI training?

Seems like there’s a fuck ton. All of Wikipedia, GitHub for code, etc.

I can understand targeting certain sites like Reddit, etc. but not random websites

It's to rip off copyrighted content and profit from it instead of the original authors. It's like every other low rent and highly automated scam that finds it's way onto the internet.

If you look closely even Google does this. This is probably why many popular sites started getting down ranked in the last 2 years. Now they're below the fold and Google can present their content as their own through the AI box.

Please remember that Google only needs to be marginally better than the competition. And, of course, their primary biz is ads, not serving great results; that is a distant second priority.

Their biz is ads, but since search is winner takes all they need only be marginally better than the competition... twenty years ago.

> Their biz is ads,

Yea, but, the FTC doesn't want it to be.

Discord I guess would be quite valuable, even the de facto public servers.

Honestly it's hard to tell how much more value the LLM people are going to get out of another copy of the internet.

It feels a lot like they're stuck for improvements but management doesn't want to hear it.

It's a bit strange to talk about stuck when the most recent breakthrough is less than a year old.

I’m not sure what you mean by breakthrough, but if you’re talking about Deepseek, it’s more of an incremental improvement than a breakthrough.

Now the only way to obtain that information is through them

I guess one could make a point that competition will no longer have the access to the scraped data.

This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations.

Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200ms to decrypt. Degrades the user experience slightly but creates a bottleneck for large-scale bots.

Check out Anubis - it's not quite what you're suggesting but similar in concept: https://anubis.techaro.lol/

I just hit a site with this --- and hit the back button immediately.

Better than a 500 error

Also better than a third party service IMO because of the privacy implications. You could even potentially give users the choice (complete overkill but technically you could do it).

Interesting! Not how I'd approach it but certainly thinking along the same lines.

Maybe once you can use something more professional as an interstitial page

From the announcement page: https://xeiaso.net/blog/2025/anubis/

> RPM packages and unbranded (or customly branded) versions are available if you contact me and purchase commercial support. Otherwise your users have to see a happy anime girl every time they solve a challenge. This is a feature.

I'm going to be making distro packages and binaries public. I will have to figure out white label monetization I guess.

You do realize that people use art like this specifically as a filter for people who react as you did. It's working 100% as intended.

> specifically as a filter for people who react as you did

A filter for normal people? A strange thing to have, but I guess it works if that's the intention.

I’m no anime fan, but this sort of judgement on normalcy leaves me with a very sour impression about whoever says this sort of thing.

It becomes a visceral reaction once you realize that a significant portion of the community is into the whole sexualizing-minors-but-nuh-uh-theyre-not-minors-they-just-look-young-because-of-their-genes ... thing. I apologize to those who are not into that, but I can't look past it.

You might say that, but claiming everyone is a pedophile is such a tired political play at this point. Its primary purpose is to dehumanize people so that blatant wrongdoing can be justified.

The visceral reaction might be genuine, but the actual feelings are probably not. I have yet to see someone who actually "cares about the children". The vast majority of accusations e.g. democrats running a pedophile ring turn out to be completely manufactured. The democrats responded by giving more funding and starting projects combating child abuse, only for the republicans to gut the programs, who think they are a waste of tax payer money and an example of big government.

I genuinely don't know how to respond to that. I'm not labeling some arbitrary political group pedophiles. I'm saying that anime has a problem where sexualization of minors is generally accepted by the community and at least a significant minority actively engages with that content. There's a plethora of articles and research on this topic. Here's a few:

[1] Examination of Sexualized Depictions of Young Schoolgirls in Anime: Opinions and Perspectives of Japanese Women - https://open.metu.edu.tr/bitstream/handle/11511/105628/Exami...

[2] Lolicon and Its Effects on Japanese Society - https://scholarsbank.uoregon.edu/items/583788d8-bbc8-4475-b1...

[3] Manga and anime: Japan still treating children as sexual objects - https://www.aljazeera.com/opinions/2014/8/11/manga-and-anime...

Violent games make people violent, and money make people greedy again?

How does that relate to anything I said?

It's not about anime itself. It's about obnoxious people who make anime their entire personality (because they have none beyond that) and shove it into everything at every opportunity.

It shouldn't upset you so much that other people express themselves and their interests without regard for your jusgement or approval.

It's open source with MIT license, so you should be able to remove those images yourself if you want.

How does it work? I don’t have time to read the code, and the website/docs seem to be under construction.

Does it have the client do a bunch of SHA-256 hashes?

Yeah it's like hashcash. You have to try random numbers until you roll a hash with enough leading zeroes. Then you get a cookie (JWT I think) that's valid for a week.

SHA2 can run on ASICs and isn't memory-hard, so I'm hoping someone will add something tougher

The problem is that the server also has to do the work. Fine for an infrequent auth challenge. Not so fine for every single data request.

Tons of problems are easier to verify than to solve.

do any of them involve mining bitcoin?

sure

Maybe there is a way for the server to ask the client to do the work?

Something similar to proof-of-work but on a much smaller scale than Bitcoin.

just add some delay to your response, we don't have to waste any more energy on meaningless calculation.

Adding delay means you have to keep more connections open at a single time. Parallelism doesn't favor a server if your problem is already a small server getting hit by a big scraper

How expensive is it to just keep a connection open?

About 20 kilobytes of socket + TLS state, if you've really optimised it down to the minimum. Most server software isn't that lean, of course, so pick a framework designed for running a million or so concurrent connections on a single server (i.e. something like Nginx)

Right it would need an algorithm with widely different encryption speeds vs decryption speeds. Lattice-based cryptography maybe?

Hash functions are all you need.

Yeah, searching for hashes with some prefix is easy to set up.

Companies running those bots have more than enough resources

Nobody has unlimited resources. Everything is a cost-benefit analysis.

For highly valuable information, they might throw the GDP of a small country at scraping your site. But most information isn't worth that.

And there are a lot of bad actors who don't have the resources you're thinking of that are trying to compete with the big guys on a budget. This would cut them out of the equation.

Make all websites intentionally waste energy as a strategy to defeat unscrupulous operators has negative costs and marginal benefits.

Then use it to mine monero or similar.

The idea that you should pay for content shouldn't be an insane pipedream. It should be the default on the internet.

Maybe then we wouldn't be in the situation where getting new users is an existential threat to the majority of websites.

Resources applied to prevent bad actors from degrading or destroying the commons has always been the cost of civilization.

I'll add a link on my site recommending that visitors petition their elected officials for a pollution tax

If we are able to detect AI scrapers then I would welcome a more strategic solution: feed them garbage data instead of the real content. If enough sites did that then the inference quality would take a hit and eventually the perpetrators, too.

But of course this is the more expensive option that can't really be asked of sites that already provide public services (even if those are paid for by ads).

I really don’t think we have such a lack of misinformation that we need to invest in creating more of it, no matter the motive.

See also https://datatracker.ietf.org/wg/privacypass/about/ and https://developers.cloudflare.com/waf/tools/privacy-pass/