A) this would only allow me to break my own stuff, so what's the risk? I just won't break my own stuff.
B) surely that's solveable with prompt engineering.
C) surely that's solveable with reinforcement training, or chaining LLMs, or <insert defense here>.
D) okay, but even so, it's not like people are actually putting LLMs into applications where this matters. Nobody is building anything serious on top of this stuff.
E) okay, but even so, once it's demonstrated that the applications people are deploying are vulnerable, surely then they'd put safeguards in, right? This is a temporary education problem, no one is going to ignore a publicly demonstrated vulnerability in their own product, right?
I’ve been exploring an LLM -> API layer for our app and I’m not worried about prompt Injection because if the user was actually malicious they could just used the interface or the API to do the same thing.
In other words if you treat the LLM like any other frontend then you really should have a problem from a security standpoint. Your would have your iOS application super user access your system, why would you treat an LLM different than any other client.
Potential vectors to consider:
- Your app lets users run it against text from other sources - fetched web pages, incoming messages - server logs - which an attacker might be able to influence
- Your users can copy and paste text into your app - and an attacker might be able to trick them into eg copying in a dozen paragraphs of text without first reading it to check for weird hidden prompt instructions
>I’m not worried about prompt Injection because if the user was actually malicious they could just used the interface or the API to do the same thing.
I think you might have missed that the injected prompt might not come from the end user.
There was an example of someone adding a prompt injection to their LinkedIn profile to override a recruiter's prompt and generate an embarrassing email instead. Not sure if it's fake, but it demonstrates the point either way.
But otherwise yeah, it can be helpful to think of prompt injection as if someone is effectively doing XSS on your AI agent (again, keeping in mind that the mitigation techniques are not the same, it's an entirely different method of attack). People tend to think of the jailbreaking examples or getting the agent to swear -- which can be embarassing but also mostly harmless. The reality is that prompt injection is basically arbitrary reprogramming of the agent, and arbitrary insertion of new tasks, and data poisoning/replacement, and data exfiltration, etc...
People who are frustrated at the safety measure that jailbreaking aims to defeat often assume prompt injection is equally "harmless" - they fail to understands that the consequences can be a lot more severe to anyone who is trying to build their own software on top of LLMs.
My insider risk — a developer who attempts to extract training data, a LLM being leaked of internal data, or an employee who wants to break the prompt for competitive gain — is a lot different of a threat than allowing all of my customers a tool to query their data using LLM’s.
https://gdpr-info.eu/art-17-gdpr/
Implementing this as a rollback-able delete will not be compliant.
[1] https://ycharts.com/indicators/ethereum_chain_full_sync_data...
Ethereum [1] routinely accumulates around 300G per year and routinely hits over one terabyte of data to sync. Remember, this is the size of the data to sync/transmit, not size of the data that is stored, which we may safely assume to be several times more, because of indices, etc.
Also, your proposal makes two tier database system: one that maintains current consistent view of the state and another for log purposes. The logging system needs high throughput storage with key range read request, which makes it, well, another pretty much fully fledged database (SELECT...GROUP BY...ORDER BY is needed).
The reason nobody does what you described because it is really prohibitive in storage space aspect and really is quite complex - a database on top of another database.
These experiences are being rushed out the door for FOMO, frenzy, or market pressure without thinking through the way people feel and what they expect and how they model the underlying system. People are being contacted for quotes and papers that were generated by ChatGPT. [1]
This is a communication failure above all else. Even for us, there’s little to no documentation.
[0] https://twitter.com/weirddalle/status/1649908805788893185
[1] https://twitter.com/katecrawford/status/1643323086450700288
Snapchat asks for the location permission through native APIs or obviously geolocates the user via IP. Either way, it’s fascinating that: people don’t expect it to know their location; don’t expect it to lie; the model goes against its own rules and ”forgets” and “gaslights.”
[0] https://www.reddit.com/r/OpenAI/comments/130tn2t/snapchats_m...
[1] https://twitter.com/somewheresy/status/1631696951413465088
If you're a high value target then it just seems like ____ aren't something you should be using
I remember when people were deciding if it was worth it to give Internet access to their internal network/users
That’s when people already had their networks and were connecting them to the internet
Eventually, people started building their networks from the Internet
But even if they’re that, they behave a whole lot more like some employee who will spill the beans given the right socially engineered attack. You can train and guard in lots of ways but it’s never “fixed.”
> The whole point of security attacks is that you have adversarial attackers. You have very smart, motivated people trying to break your systems. And if you’re 99% secure, they’re gonna keep on picking away at it until they find that 1% of attacks that actually gets through to your system.
If you're a high value target then it just seems like LLMs aren't something you should be using, even with various mitigations.
And somewhat related to that, the purpose of the system should be non-destructive/benign if something goes wrong. Like it's embarrassing if someone gets your application to say something horribly racist, but if it leaks sensitive information about users then that's significantly worse.