You're confusing prompt injection with jailbreaking. The danger of prompt injection is that when your GPT tool processes 3rd-party text, someone else reprograms its instructions and causes it to attack you or abuse the privileges you've given it in some way.
> spreading basically FUD around GPT
My impression is that Simon is extremely bullish on GPT and regularly writes positively about it. The one negative that Simon (very correctly) points out is that GPT is vulnerable to prompt injection and that this is a very serious problem with no known solution that limits applications.
If that counts as FUD, then... I don't know what to say to that.
If anything, prompt injection isn't getting hammered hard enough. Look at the replies to this article; they're filled with people asking the same questions that have been answered over and over again, even questions that are answered in the linked presentation itself. People don't understand the risks, and they don't understand the scope of the problem, and given that we're seeing LLMs wired up to military applications now, it seems worthwhile to try and educate people in the tech sector about the risks.
Prompt injection is more like a "cheat" code - yeah, you can "noclip" through walls, but you're not going to get the ESL championship.
I don't understand what you mean by this. LLMs are literally being wired into military applications right now. They're being wired into workflows where if something falls over and goes terribly wrong, people actually die.
If somebody hacks a Twitch bot, who cares? The problem is people are building stuff that's a lot more powerful than Twitch bots.
Do you have any proof to back this claim?
What do you think happens if that AI starts lying about what units are available or starts returning bad data? Palantir also mentions wiring this into autonomous workflows. What happens when someone prompt injects a military AI that's capable of executing workflows autonomously?
This is kind of a weird comment to be honest. I want to make sure I understand, is your assertion that prompt injection isn't a big deal because no one will wire an LLM into a serious application? Because I feel like even cursory browsing on HN right now should be enough to prove that tech companies are looking into using LLMs as autonomous agents.
Of course the application-infrastructure might be vulnerable as well in case the user IS the attacker, but it's more difficult to imagine concrete examples at this point, at least for me.
> This is crucially important. This is not an attack against the AI models themselves. This is an attack against the stuff which developers like us are building on top of them.
That seems more like a community service, really. If you're building on the platform it's probably a relief to know somebody's working on this stuff before it impacts your customers.
The concern is that people building people-facing or people-handling automation will end up putting their abstractions on the road before inventing seatbelts — and waiting for a Volvo to pop up out of mushrooms isn’t going to be enough in case haste leads to nuclear waste.
It is a policy issue as much as it is an experience issue. What we don’t want is policymakers breaking the hammers galvanized by such an event. And with Hinton and colleagues strongly in favor of pauses and whatnot, we absolutely don’t want to give them another argument.
...and my last worry is people subverting the prompt to ask "stupid" questions - I send the prompts to a moderation API and simply block invalid requests.
Folks, we have solutions for these problems and it's always going to be a cat and mouse game.
"There is no such thing as perfection" (tm, copyright and all, if you use this quote you have to pay me a gazzilion money)
But people are wiring GPT up to real-world applications beyond just content generation. Summarizing articles, invoking APIs, managing events, filtering candidates for job searches, etc... Greshake wrote a good article summarizing some of the applications being built on top of LLMs right now: https://kai-greshake.de/posts/in-escalating-order-of-stupidi...
Prompt injection really heckin matters for those applications, and we do not have solutions to the problem.
Perfection is the enemy of the good, but sometimes terrible is also the enemy of the good. It's not really chasing after perfection to say "maybe I don't want my web browser to have the potential to start trying to phish me every time it looks at a web page." That's just trying to get basic security around a feature.
Given the allure of using AI in the military for unmanned systems it’s not that far off.
With a lesser danger level, similar adversarial dynamics exist in other places where AI might be useful. E.g dating, fraud detection, recruitment
Is GPT perfect? Hell, no?
Does it have biases? F*c yeah, the same ones of the humans that programmed it.
This might be technically correct, in the sense that I think these companies have their own LLMs they're pushing? They're not literally using OpenAI's GPT model. But all LLMs are vulnerable to this, so it doesn't practically matter if they're using specifically GPT vs something in-house, the threat model is the same.
It's like an SQL-injection without a commonly accepted solution. And that's why it's a serious issue.
I know how to handle potential SQL-injection now. And if I don't I can just google it. But were I that informed when I wrote the first line of code in my life? Of course not.
Now the whole world is just as ill-informed about prompt injection as I were about SQL-injection by the time.
I suggest reading my blog closer if you think I'm trying to scare people off GPT. Take a look at these series of posts for example:
https://simonwillison.net/series/using-chatgpt/ - about constructive ways to use ChatGPT
https://simonwillison.net/series/llms-on-personal-devices/ - tracking the development of LLMs that can run on personal devices
See also these tags:
- llms: https://simonwillison.net/tags/llms/
- promptengineering: https://simonwillison.net/tags/promptengineering/
You've also seen a bunch of my content on Hacker News because I'm one of the only people writing about it - if very much like not to be!
With all due respect, I would also like to market someone else who has also been posting similar content, but for some reason those posts never make it to the top. If you don't believe me, you can check the following submissions:
[0]: https://news.ycombinator.com/item?id=35745457
[1]: https://news.ycombinator.com/item?id=35915140
They have been consistently putting the risks of LLMs. Thanks for spreading the information though. Cheers.
I see GPT as a tool to make "my life easier", help me with tedious stuff, maybe point out some dark corners etc
Why would I go and try to break my hammer when I need it to actually put the nails in?
Will there be users doing that? Sure!
Will I be doing that?
Not really, I have real issues to take care of and GPT helps do that.
Maybe I'm missing something, but this is more like sql-injection with php/mysql - yes, it's an issue and yes, we need to be aware of it.
Is it a "nuclear bomb"-type issue?
I would say no, it isn't.
#off-topic: I counted at least 4 links (in the past 2 weeks!) to Simon's website for articles spreading basically FUD around GPT. Yes, it's a new technology and you're scared - we're all a bit cautious, but let's not throw out the baby with the bathwater, shall we?