> This idea of software threatening the user with consequences is totally wild and dystopian.

This idea of software built on top of reverse-engineered data threatening the user with consequences is what's really even wild and dystopian.

god you're so right

All sorts of tools try to prevent dangerous/destructive uses

In fact probably every single piece of commercial software you use had you sign a contract saying you wouldn’t do it

> All sorts of tools try to prevent dangerous/destructive uses

But they don't threaten their users or have an "N strikes and you're out" policy. I take those safety caps off of all the chemicals in my garage because I'm a grown-ass adult and those caps are a pain in the butt. I would not expect the manufacturer of a solvent to show up at my house lecturing me about safety and threatening to ban me from buying his products.

Sure but they would if they could. If they knew idiots were doing idiot things with their products (or evils doing evil things) and did not utilize available methods to prevent them, then the company ends up holding liability. And no, this is not easily signed away in a contract.

There actually is a very important distinction between "would if they could" and "they can and do", though.

Uhh right, but describing that as "dystopian" is frankly hysterical.

It's an obvious corollary of good things (like product liability). Virtually everyone I've heard complain about these safety rails was up to antisocial (at best) stuff. I've never heard a sympathetic use-case. It's objectively good that companies can be held responsible for misuse of their products and that they are therefore incentivized to mitigate misuse.

"My inability continuously attack product guardrails to enable my super esoteric (and probably antisocial) use-case is dystopian" is just... not a compelling argument.

Yes, my safety cap policy is definitely anti-social.

"These safety rails" was referring to LLMs, which have far more nuanced and capable safety rails than chemical caps do, and accordingly also have much more assertive ways to enforce them.

It's the same underlying principle. If I want to ask a software tool what the suicide rate is for my county, I do not expect it to come back with: "Naughty boy! You said an unsafe word! You're getting a strike, and if you get two more, you're banned." This is totally out of the ordinary for a software product, and is absolutely a modern invention. Replace "suicide" with whatever the "AI Safety" obsession word is today.

> If I want to ask a software tool what the suicide rate is for my county, I do not expect it to come back with: "Naughty boy! You said an unsafe word! You're getting a strike, and if you get two more, you're banned."

Did this happen?

I just tested this query in Grok, Gemini, Claude, and ChatGPT and 0% of them admonished me or refused to return an answer.

Just like every single conversation I've ever had on this topic, you have to make up examples that aren't even true. Why don't you just share what you were doing that you feel you were unfairly prevented from?

(I have an inkling why you won't do that...)

That's why I said:

> Replace "suicide" with whatever the "AI Safety" obsession word is today

I don't know what those queries are, but original-OP made one and got a "strike", which is what spawned this thread.

Which would be more than 0% concerning if I've ever heard (even once) an example of this happening with a query that shouldn't actually trigger something like that, or is so close to such a query, that the false positive is understandable and of incredibly niche value anyway.

OP gave an example of reverse engineering, something that to the LLM looks identical to just hacking. I am totally fine if the incredibly tiny little fraction of people who want to reverse engineer their own systems can't use LLMs to do it, and in exchange top LLMs aren't helpful for the hordes of actual malicious actors who would love a superintelligence to aid their crimes.

No-brainer tradeoff, just like 100% of examples I've ever heard.

I don't think that "dystopian" necessarily goes far enough, this would be one of the rare times where I would call it a fascist mentality - the idea that everything's primary allegiance is to the state and the goals of the state rather than those of the customer or the user.

I want a default that has people empowered, rather than something where it's just another performative smokescreen caused by overzealous product liability. I'll thank you and your kind for needing to distractedly tap the "Agree" button on my car's infotainment every time I start it to confirm that I will pay attention to the road.

"the state" is just shorthand we use for "other people in my community"

> I'll thank you and your kind for needing to distractedly tap the "Agree" button on my car's infotainment every time I start it to confirm that I will pay attention to the road.

Does that actually mitigate antisocial usecases? No? Then it's not what I'm talking about :)

Of course if you wanted to you could just share specifically what totally-reasonable LLM use-case you have in mind that's neutered by this "fascist mentality" instead of dreaming up unrelated instances.

> "the state" is just shorthand we use for "other people in my community"

It's a very different abstraction layer, in the same way as individual cells vs the entity that is you. The entity that comes together from all those "other people in my community" and its priorities are different to the individual desires.

> Does that actually mitigate antisocial usecases? No? Then it's not what I'm talking about :)

Maybe it does? Maybe someone is alive on the road today because they read the message and changed their behaviour. I'm giving an example of something where this liability mindset has created a world where manufacturers are no longer prioritising the desires of their users in order to appease a sense of harm-reduction. And you weren't limiting it to LLMs you were applying it to all sorts of tools.

I think that "reverse engineering" as the OP was talking about is one of those things where maybe 1/10000 uses could actually be harmful. This is not even a high-risk request such as to produce a weapon of some kind where maybe your "antisocial usecases" could be applied.

Yes if you apply some logic to such extremity that it produces bad outcomes then you should stop applying that logic to those extreme cases.

I think it's closer to asking a remote (human) assistant to do something that someone doesn't want done (e.g., view the source of a closed-source product, whether through reverse engineering, going into their office, or social engineering) and that remote assistant company saying, "Please stop asking our assistants to do that."

You can still use an IDE (hammer) to reverse engineer anything you want.

It's not though. It's still just a piece of code, much closer to IDEs or any other program than to a human assistant in any way that matters (morals, responsibility).

It just seems like you are saying if you found out Claude code was a bunch of remote working doing work for you, then it would be morally wrong to do illegal/morally wrong/irresponsible things with them, but because it is NOT a human, those same things are fine?

Yes, correct.

Is the distinction between human labor/actions and a program executing hard to grasp?

Moral is a human thing, not an absolute thing, so of course it's different if there is a single human involved and a tool, and a human with a relationship to other humans.

I just have different moral preferences. I think its morally wrong to do illegal/morally wrong/irresponsible things in general, whether I am using a hammer, a car, a company, or AI.

It's worse to ask someone else to break a window with a hammer for me, but the window still got broken, and the person whose window it was is still sad/out of money/etc.

The thought experiment was that if you were doing illegal things with an AI, you would not feel bad, but if you found out that the AI was a person, you would feel bad. That is very strange to me, more a feeling of guilt/shame.