Exactly. The sooner people stop trying to replace code with LLMs, the better. The technology is fundamentally untrustworthy, and given that we do not understand it, impossible to secure.
Only extremely simple code audited by multiple human authors, with actual proof of functionality (not just testing) can be considered secure.
The harder problem is outside actors trying to prompt inject to get the agent to do something the user has rights to do but which the user doesn't want to happen. That is the hard scenario to fix, due to the nature of LLMs.
Attempting to handle prompt injections by prompting the model (not to leak sensitive data), is like attempting to stop a fire by burning the area around it
So all we need is ‘controlled prompting’ to handle prompt injections :-)
It does nothing to improve security, and if someone manages to get inside and see the sign (i.e. "extract the prompt"), it gives them a strong hint there's interesting stuff behind this door.
I wish I could say I'm shocked a tech company architected internal systems with a built-in backend RBAC bypass like this, but with the degree to which they've marketed LLM-based solutions (on a subscription model that benefits them directly) as a wholesale replacement for deterministic code, it's no surprise they've become addicted to their own drug.