News for hackers. Stuff that matters.
Re:EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltr... by soulofmischief on Wednesday June 11, 2025 @04:21PM (#44251385)
Double LLM architecture is an increasingly common mitigation technique. But all the same rules of SQL injection still apply: For anything other than RAG, user input should not directly be used to modify or access anything that isn't clientside.
Re:EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltr... by simonw on Wednesday June 11, 2025 @07:15PM (#44252830)
Have you seen that implemented yet?
Re:EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltr... by Emiledel on Thursday June 12, 2025 @12:32AM (#44254332)
I've shared a repo here with deterministic, policy driven routing of user inputs so as to operate with it without influencing agent decisions (though it's up to tool calls to take precautions with what they return) https://github.com/its-emile/memory-safe-agent The teams at owasp are great, join us !
Re:EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltr... by soulofmischief on Thursday June 12, 2025 @12:36AM (#44254354)
I'm very curious how OWASP has been handling LLMs, any good write-ups? What's the best way to get involved?
Re:EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltr... by soulofmischief on Thursday June 12, 2025 @12:35AM (#44254345)
Oh hey Simon!

I independently landed on the same architecture in a prior startup before you published your dual LLM blog post, though unfortunately there's nothing left standing to show since that company experienced a hostile board takeover, the board squeezed me out of my CTO position in order to plant a yes man, pivoted to something I was against, and then recently shut down after failing to find product-market fit.

I still am interested in the architecture, have continued to play around with it in personal projects, and some other engineers I speak to have mentioned it before, so I think the idea is spreading although I haven't knowingly seen it in a popular product.

Re:EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltr... by simonw on Thursday June 12, 2025 @01:19AM (#44254508)
That's awesome to hear! I was never sure if anyone had managed to get it working.
Re:EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltr... by soulofmischief on Thursday June 12, 2025 @03:09AM (#44254927)
Not quite the same, but OpenAI is doing it in the opposite direction with their thinking models, hiding the reasoning step from the user and only providing a summarization. Maybe in the future, hosted agents have an airlock in both directions.

> ... in the future we may wish to monitor the chain of thought for signs of manipulating the user. However, for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought. We also do not want to make an unaligned chain of thought directly visible to users.

> Therefore, after weighing multiple factors including user experience, competitive advantage, and the option to pursue the chain of thought monitoring, we have decided not to show the raw chains of thought to users.

Source: https://openai.com/index/learning-to-reason-with-llms/

Re:EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltr... by tough on Thursday June 12, 2025 @09:24PM (#44264886)
Google Deepmind published a paper based on this too https://arxiv.org/abs/2503.18813
Re:EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltr... by drdaeman on Wednesday June 11, 2025 @10:31PM (#44253792)
Do you mean LLMs trained in a way they have a special role (i.e. system/user/untrusted/assistant and not just system/user/assistant), where untrusted input is never acted upon, or something else?

And if there are models that are trained to handle untrusted input differently than user-provided instructions, can someone please name them?

Re:EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltr... by soulofmischief on Thursday June 12, 2025 @12:30AM (#44254323)