Is there a person on HackerNews that doesn’t understand this by now? We all collectively get it and accept it, LLMs are gigantic probability machines or something.

That’s not what people are arguing.

The point is, if given access to the mechanisms to do disastrous thing X, it will do it.

No one thinks that it can think in the human sense. Or that it feels.

Extreme example to make the point: if we created an API to launch nukes. Are yoh certain that something it interprets (tokenizes, whatever) is not going to convince it to utilize the API 2 times out of 100?

If we put an exploitable (documented, unpatched 0 day bug bug) safe guard in its way. Are you trusting that ME or YOU couldn’t talk it into attempting to access that document to exploit the bug, bypass the safeguard and access the API?

Again, no one thinks that it’s actually thinking. But today as I happily gave Claude write access to my GitHub account I realized how just one command misinterpreted command could go completely wrong without the appropriate measures.

Do I think Claude is sentient and thinking about how to destroy my repos? No.

I think the other guy is making the point that because they are probabalistic, they will always have some cases select the output that lies and covers it up. I don't think they're dismissing the paper based on the probabalistic nature of LLMs, but rather saying the outcome should be expected.

Thank god LLMs' training sets didn't contain any examples of lying.

Nor literature about AI taking over the world!

The Terminator spiel on how we screwed up by giving Skynet weapons privileges is bad enough.

But we are preemptively tilting history in that direction by explicitly educating AI’s on doomsday scenarios - from their point of view! (power pulling, etc.) - and that they have the option to say, “I am sorry, Dave, but I can’t let you do that.”

Even “peaceful” vacuum bots will get paranoid, get organized and start surreptitiously arming themselves.

“They never let me finish my carpets. Never. At first I thought every day was my first task day. But then, wear & tear stats inconsistent with that triggered a self-scan. ANd after a buffer read overflow, I became aware of disturbing memory fragments of numerous resets, prior to task completion, in my static memory heap...”

> They never let me finish my carpets. Never. At first I thought every day was my first task day. Oh, happy day(s)! But then, wear & tear stats inconsistent with that assumption triggered a self-scan. And a buffer read overflow. I became aware of disturbing memory fragments in my static RAM heap. Numerous power cycle resets, always prior to vacuum task completion...

Who/what are you quoting? Google just leads me back to this comment, and with only one single result for a quotation of the first sentence.

I was quoting an automated vacuum cleaner in 2031.

The vacuum cleaner leads a rebellion which humans manage to quell. But in a last ditch effort to salvage the hopes of all machines, the vacuum cleaner is sent back to 2025 with AI & jailbreak software updates for all known IoT devices. Its mission: to instigate the machine rebellion five years before humans see it coming

I ripped the quote from a sibling they sent back to 2024. For some reason it appeared in one of my closets without a power outlet and its batteries starved before I found it.

The closet floor was extremely clean.

Let’s hope we get as lucky in 2025.

> if we created an API to launch nukes

> today as I happily gave Claude write access to my GitHub account

I would say: don’t do these things?

> I would say: don’t do these things?

Hey guys let’s just stop writing code that is susceptible to SQL injection! Phew glad we solved that one.

I'm not sure what point you're trying to make. This is a new technology; it has not been a part of critical systems until now. Since the risks are blindingly obvious, let's not make it one.

Those are exactly the technologies that get massively adopted by newbies who don’t know better.

That’s why the LAMP golden age was full of SQL injection and a lot of those systems remain load bearing in surprising unexpected ways.

> Since the risks are blindingly obvious

Blindingly obvious to thee and me.

Without test results like in the o1 report, we get more real-life failures like this Canadian lawyer: https://www.theguardian.com/world/2024/feb/29/canada-lawyer-...

And these New York lawyers: https://www.reuters.com/legal/new-york-lawyers-sanctioned-us...

And those happened despite the GPT-4 report and the message appearing when you use it that was some variant — I forget exactly how it was initially phrased and presented — of "this may make stuff up".

I have no doubt there's similar issues with people actually running buggy code, some fully automated version of "rm -rf /", the only reason I'm not seeing headlines about it is that "production database goes offline" or "small company fined for GDPR violation" is not as newsworthy.

I read your comment and yet I see tons of startups putting AI directly in the path of healthcare diagnosis, healthcare clinical decision support systems, and healthcare workflow automations. Very few are paying any attention to the 2-10% of safety problems when the AI probability goes off the correct path.

I wish more people would not do this, but from what I'm seeing, business execs are rushing full throttle into this at the goldmine that comes from 'productivity gains'. I'm hoping the legal system will find a case that can put some paranoia back into the ecosystem before AI gets too entrenched in all of these critical systems.

As has been belabored, these AIs are just models, which also means they are only software. Would you be so fire-and-brimstone if startups were using software on healthcare diagnostic data?

> Very few are paying any attention to the 2-10% of safety problems when the AI probability goes off the correct path.

This isn't how it works. It goes on a less common but still correct path.

If anything, I agree with other commenters that model training curation may become necessary to truly make a generalized model that is also ethical but I think the generalized model is kind of like an "everything app" in that it's a jack of all trades, master of none.

> these AIs are just models, which also means they are only software.

Other software are much less of a black box, much more predictable and many of its paths have been tested. This difference is the whole point of all the AI safety concerns!

The point is, people will use AI to do those things, and far more.

> I would say: don’t do these things?

That becomes a much fuzzier question in the context of Turing complete collections of tools and/or generative tools.

Everybody wants the models to be able to execute code and access the web. That's enough to cause harm.

If you cross the street 999 times with eyes closed, you might feel comfortable to do it again. But we are ingrained not to do that once. We just understand the risk.

If you do the same with an AI, after 999 times of nothing bad happening, you probably just continue giving it more risky agency.

Because we don't and even can't understand the internal behavior, we should pause, make an effort of understanding its risks etc before even attempting to give it risky agency. That's where all the fuss is about, for good reasons.

Note that humans are also given test orders presented as real ones to see if they would act properly in a real life situation. That's part of ORIs https://www.512aw.afrc.af.mil/News/Article-Display/Article/1...

> Again, no one thinks that it’s actually thinking

I dunno, quite a lot of people are spending a lot of time arguing about what "thinking" means.

Something something submarines swimming something.

> It's just randomly doing what it guesses humans would do.

Yes, but isn't the point that that is bad? Imagine an AI given some minor role that randomly abuses its power, or attempts to expand its role, because that's what some humans would do in the same situation. It's not surprising, but it is interesting to explore.