thesneakykoala

But does it need to know ahead of time or is it enough to be able to test 10k different directions at once then analyze the results? I have thought the same way in the past, that LLMs can't innovate due to how they are trained, but many of humanity's most important discoveries (fire, antibiotics, microwaves, etc) were discovered accidentally in the pursuit of something else. Even if the AI can't truly invent, all it needs to do is generate and test hypotheses then analyze to see if there is and there there for a human to check out.

Maybe we should train them to be evil in RL so that we can better see the "evil" activations. Training them to be "good" in RL would seem to cause the aurpression of misaligned characteristics.

For example, when we consider the recent PHP issues : the current generation of AI models would have ingested these flaws in pretraining data and might even have been brought through reinforcement learning trials where the flaw came up.

So what happens in a reinforcement learning trial if there's an exploitable bug in a language that's being used for the trial?

A) the AI doesn't mention the security flaw and passes the test. Branch survives.

B) the agent raises an issue with a bug in the language that has a security issue. This isn't a valid answer for the test, so this branch is pruned.

C) the agent pokes at the bug or even starts an exploit, is caught by a monitor or ultimately fails the benchmark, and is pruned (surpressing some of the characteristics that lead to its failure).

So it seems reasonable that our current models could have been exposed to a ton of vulnerable code in pre training that's currently in production and then we could have suppressed 1) its tendency to express its awareness of these flaws, and 2) the characteristics that caused it to fail when investigating these vulnerabilities as a side task during RL trials.

(I wrote the above section a couple weeks ago)

If we only train them on "good" behavior, then we may minimize "evil" activations before we identify them and then cause them to be difficult to detect. If we continue in this direction, we may just suppress its "ethically problematic" introspection, in which case we would lose one of the only indicators they point out in this blog.