Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs [pdf]

Re:Emergent Misalignment: Narrow finetuning can produce broa... by Terr_ on Tuesday February 25, 2025 @07:53PM (#43179417)

> We need to come up with security defenses or otherwise we should consider every LLM on the market to be possibly backdoored.

My rule-of-thumb is to imagine that the LLM is running client-side: It can't reliably keep a secret, and a dedicated user can force it to an output of their choosing.

The problem can't be solved with "prompt engineering", because the underlying algorithm is just a Make Document Longer machine and there's no fundamental distinction between the real prompt versus a user stating something versus the LLM repeating something as requested.

[ Parent | on HN ]

Re:Emergent Misalignment: Narrow finetuning can produce broa... by mlyle on Wednesday February 26, 2025 @12:49AM (#43181037)

I don't think this is the threat they're talking about.

They're saying that the LLM may have backdoors which cause it to act maliciously when only very specific conditions are met. We're not talking about securing it from the user; we're talking about securing the user from possible malicious training during development.

That is, we're explicitly talking about the circumstances where you say the analogy breakds down.

[ Parent | on HN ]