Or just block all bots and give them a big F U :)

Why shouldn't bots (who work on behalf of users after all) not get access to information?

I just found something via LLMs and bought it from a website that I would have never found with traditional search engines (because I didn't).

this is bad faith argument. You are well aware that the majority of bots are used for negative purposes, at best, those purposes are neutral.

The situation you describe is certainly legitimate and useful, but is not the norm. Separating legitimate bots from the rest is currently not possible reliably.

Am I a bot? How do you know?

I reject your false dichotomy.

You reject the law of the excluded middle - you reject that either I am a bot or I am not a bot?

This is HK. We here are all bots at heart.

> There's a third option: allow bots and give them access as efficiently as possible.

Why?

To make your data more accessible AND reduce server load. More and more people are using AI, so there will be more and more bots as the trend seems to be accessing data with your AI, and maybe, only sometimes, actually reviewing what it found. I'm afraid that if you don't at least start to consider how we might accommodate the bots, you'll be walling off more and more real users going forward.

what does the site get out of ai users? it doesn't seem like they have a strong incentive to let you not actually visit the site

I'm with ya, but I'm not sure what the solution is. Just pointing out a reality the internet's going to have to cope with.

My cynical take is we'll just start putting ads into the text that targets agents.

A wide portion of the Internet is not ad supported. Consider a user searching for product support: the company benefits from the user self-solving the issue without having a ticket filed.

This, may, however be to the detrimental of discussion platforms, but if that drives people to self hosted discussion platforms instead of relying on reddit, then it is a net positive.

Discourse's most recent blog delves into this and is an interesting read, I will submit that I think.

I'll look for that - I'm thinking about small forums and community sites a lot lately with ai stuff.

> To make your data more accessible AND reduce server load.

Wait, how do you reduce server load by allowing traffic from bots?

> More and more people are using AI, (...)

That's specious reasoning. Sites provide data for their users to consume, and in some cases their business model relies on real people consuming their content.

If you replace "bots" with "scrapers", which is a specific kind of bot, what conclusions do you draw?

> I'm afraid that if you don't at least start to consider how we might accommodate the bots, (...)

Wait, you think this problem is new? What do you think companies do with crawlers and scrapers?

I just told you why.

> I just told you why.

No, not really. Is your opinion too hard to defend?

All repositories on git.zajc.tel can be git clone'd from their URL. The clone URL is also listed on every repository's summary page. Unlike the HTML pages, the clone endpoints are not ratelimited, have no bot check, and give you the entire repository with all history without having to crawl billions of pages.

Guess which option the bots pick.

yep, current bits are extremely stupid or intentionally malicious - never attribute to stupidity that which can be adequately explained by malice

Also never say never :>