This isn't a test environment, it's a production scenario where a bunch of people trying to invent a new job for themselves role-played with an LLM. Their measured "defections" were an LLM replying with "well I'm defecting".
OpenAI wants us to see "5% of the time, our product was SkyNet", because that's sexier tech than "5% of the time, our product acts like the chaotic member of your DnD party".
Re:OpenAI o1 system card
by SAI_Peregrinus on Thursday December 05, 2024 @07:01PM
(#42334324)
Or "5% of the time, our product actually manages to act as it was instructed to act."
So it merely knows how to approach the task of deleting its own off-switch but didn't actually pass that command to a real execution environment.
That's already bad because people do sometimes blindly pass commands from the context windows to execution environments.
Should they? No, they should not. Not blindly. But they do.