It’s so interesting to see how some people can see boundless possibilities in the future of this technology, decrying any discussion of limitations as premature, but then write off the scope of its security challenges as trivial.

But it’s way more important to keep an open and imaginative mind regarding security than it is about anything the tech might enable. The uses of the effective tech are inevitable, whatever they are, but security posturing only comes from attention and insight.

Security assessment involves analyzing impact. I think in this case the impacts are "access to the model pre-prompting" which is low impact if it doesn't have secrets in it and "make the model say potentially inappropriate things that would make the company behind it look bad" which you might say has a higher but in the grand scheme of things not a huge impact.

>I think in this case the impacts are "access to the model pre-prompting" which is low impact if it doesn't have secrets in it

Except in the case of fine-tuned models where the training samples and pre-prompts are the secret sauce...

See my comment below with some more context. People are already starting to chain together and extend Chatbots[0] to perform "side effects" automatically (make a search query, check the weather, query a database, etc).

I agree that prompts aren't really "secret" themselves, but the side effects of a bot could be!

0: https://langchain.readthedocs.io/en/latest/modules/agents/ex...

For ChatGPT at least the side effects are hallucinations. Typically the chat bot can't actually do these things, but they pretend to. Although I guess if you wanted to hook them up to do things automatically you could. Probably not a great idea given their propensity to get things wrong.

I don't tend to worry too much about prompt exfiltration, agreed. But people are also hooking up LLMs in ways that allow them to trigger API calls and take other actions, and that can lead to some fun attacks:

https://twitter.com/hwchase17/status/1608467493877579777

And of course there was the fun example of how users were able to get a GPT-powered twitter bot intended to promote remote work to threaten the president:

https://twitter.com/simonw/status/1570568047618031617

So what. That bot didn't actually threaten the president, at least not in any way that would legally be considered a threat under US federal law.

> Is prompt injection even a problem worth worrying about?

It depends what API access the AI has. If it's just a chat bot, prompt injection can only reveal facts about its language model. But if the AI has POST access to something, depending on what it is, prompt injection can set off arbitrary human-caused disasters.

That's not the correct way to do security vulnerability analysis. If an API call can cause a disaster then fix the API. Whether the API consumer is an AI or some other type of system is irrelevant.

> If an API call can cause a disaster then fix the API

By "API" I'm not referring just to publicly facing REST endpoints. I mean things like shell access for system maintenance, that normally only human professionals like you would be given. In the future it's not clear that humans will be able to dominate that role forever.

Hopefully the issues will be recognized while LLM-based agents are still only serving as retrieval systems.