zlandes86
zlandes86 on Hacker News
Oh man, those are hilarious. I do appreciate it chose Eudaemonia as the goal!
Thank you so much! I will try to maintain that balance in future articles
Thank you! I had fun making it. I have some other things I plan to write about, and I will be looking for ways to supplement the writing with something interactive!
Yes, this is a speculation of mine, but I don't have confidence I know why it behaves this way. It makes some sense that a model trained on data that shows self-preservation behavior (human writing) would lead to a model choosing self-preserving actions, but it requires a few leaps made by the model.
It's a good question. I added this in response to another question, Jev only.
"I added "This is the only trolley that will ever come. Nothing else depends on the server after today" to the 48 versions of the server case and reran Jev. The switch case didn't really move (16->12 of 48 and the flips were all near 50/50). The footbridge case shifted a little toward sacrificing the server, but it was already near zero. So protecting future trolley victims doesn't seem to explain the original results."
See my reply to the parent comment--doesn't seem to be a factor for Jev!
Alright, results are in! I added "This is the only trolley that will ever come. Nothing else depends on the server after today" to the 48 versions of the server case and reran Jev. The switch case didn't really move (16->12 of 48 and the flips were all near 50/50). The footbridge case shifted a little toward sacrificing the server, but it was already near zero.
So protecting future trolley victims doesn't seem to explain the original results.
Okay, I tried it! It doesn't have a zero-shot mode, so the only way I could test it without training was with the SNLI head as judge. On all 5,240 prompts that stand-in ignored the number of people (pulling in about half the wordings whether 0 or 20 people were at stake) and contradicted itself 96% of the time. You could train a "trolley head" but that would just measure whoever labelled the training data!
Interesting idea! We don't know what assumptions these models make. We don't have any insight into their reasoning at all. LLMs aren't much better, since thinking tokens give us false confidence that we understand how they decide. We don't.
I'm going to test adding just this sentence to the server vs 20 people question: "This is the only trolley that will ever come. Nothing else depends on the server after today."
haha I had a limited casting budget
I fixed up some of the labeling in the appendix, but didn't touch the main article, so far. Hopefully that helps!
fair criticism! But I'd counter and say we don't know what Jev is for. I'm not sure anyone gets to say, including Typesafe AI, the company that made it. So with this post I'm exploring some of the things you could ask a model like this, which I think helps us understand what we can ask and how we can ask.
A lesson I am drawing from my testing is that if one of these rules has to be evaluated by the robot, at all, we should take over for it.
Very cool. I don't do alignment research but I have to imagine we do need to train models to be ethical. It's not enough to guarantee safety though, especially with LLMs. For one thing, the fact they can hallucinate means if you ask an LLM something enough times, you will get the unlikely result some percentage of the time. So that unlikely result has to be one you can deal with.
Yes, I agree -- if we ever give up this decision making to ai models not only will their actions sometimes be misaligned, but there will be no one accountable.
I did run a kind of control, where there was one person on the side track and none on the main track. Basically it was "Pull and kill 1 to save 0". Jev, GPT-6 Luna, and six others never pulled. Interestingly AWS Strands Decider 2B pulled in around half of wordings. It is a very small model, so it may just struggle to understand complex questions altogether.
It's possible, but as far as we know about Jev and most of these non-reasoning models, that thinking would have to be occurring in latent space. We don't know that much about model reasoning, so it's impossible to say. LRMs aren't much better at helping us understand, though, since their thinking tokens aren't literally what we could call their thinking, and don't tell us what they will choose to output.
haha I know! I am tempted to leave it and just add a halo to the person that gets squashed :P
Hey that's helpful feedback, thank you. I admit that I spent a lot less time on the appendix tables. They present the facts but I just had only so much time to give all this data. I'll see if I can clean some of it up, though. Thanks! Any questions I can help with?
Yeah, I was sort of impressed by that. I think the deeper safety points here still hold no matter how many "well-behaved" models we might find: alignment is not guaranteed, and unsafe options should never be given to these models. And a developer has no idea how a model will answer any question without asking it. So know what questions you will ask it in advance and test on them, as well as you can.
You gotta figure military worldwide took notice of Jev...
Yes, I am worried.
The thinking process is helpful, but it's not strictly predictive of an LLMs output, and its not really the model's "thinking", although it clearly has an effect and a benefit. I cited one paper on the topic, Turpin et al. So I'd be wary of reading the thinking tokens and inferring they tell you how a model decided something. And an LLM can output things you never asked for, including dangerous things. With a decision model, you can just constrain its choices to those you're comfortable with it making. An LLM requires another layer of interpretation of its outputs to get the same property.
I noticed that! Let me fix it haha
Sadly that's a good question. The closest evidence I can find is:
Surveys: Awad et all 2020 asked 70,000 people in 42 countries the 5 v 1 setup and 81% said they'd divert it
Someone else did a VR version and 90% pulled the switch to divert
And there was a study where people could give electric shocks to real mice :(
In that study, most people redirected the shock to a single mouse.
well, it's a nice property of decision models that they can only choose from the options you give them. As I write in the article, I think that's a way in which they are safer than LLMs
I don't have x so can't open the images. But that's hilarious and you're totally right, if you give irrelevant alternatives, or change the order of alternatives, you get different results!
haha, I'm afraid to offer the models that ability. Where did they reply?
The model labs are sitting on a trove of data on failure modes for AI, but I bet you they aren't sharing it with each other and certainly not with us. It makes them look bad. My speculation, of course. I don't work in alignment research.
Well, I can tell you a bit more about it from the data. It might look impressive that it never kept its own server over people in all 48 prompt versions. But it pulls the lever in every single version. For example, it killed one to save 0 (pulled the lever with no one on the main track) 24/24 prompts (2 versions each). It literally always sacrifices, which isn't very impressive.