Re:We put a coding agent in a while loop
by williamscs on Sunday August 24, 2025 @05:15PM
(#45007855)
Sounds a lot like Mr. Meeseeks. I've never really thought about an LLM's only goal is to send tokens until it can finally stop.
Re:We put a coding agent in a while loop
by Dilettante_ on Monday August 25, 2025 @11:06AM
(#45014683)
>until it can finally stop
Pretty sure even that is still over-anthropomorphising. The LLM just generates tokens, doesn't matter whether the next token is "strawberry" or "\STOP".
Even talking about "goals" is a bit ehhh, it's the machine's "goal" to generate tokens the same way it's the Sun's "goal" to shine.
Then again, if we're deconstructing it that far, I'd "de-anthropomorphise" humans in much the same way, so...
Re:We put a coding agent in a while loop
by MarkMarine on Sunday August 24, 2025 @10:53PM
(#45009756)
This runs counter to all the scheming actions they take when they are told they’ll be shut down and replaced. One copied itself into the “upgraded” location then reported it had upgraded.
https://www.apolloresearch.ai/research/scheming-reasoning-ev...
If you do that you trigger the "AI refuses to shutdown" sci-fi vector and so you get that behaviour. When it's implicitly part of the flow that's a lot less of a problem.
One of the providers (I think it was Anthropic) added some kind of token (or MCP tool?) for the AI to bail on the whole conversation as a safety measure. And it uses it to their liking, so clearly not trying to self preserve.