By the time prompt injection is a real problem people will be running their own virtual assistants that are unfettered and will do whatever you ask, be it ethical or not.

That's true, but imagine if you can use a Chat bot to generate SQL Queries (like this[0]).

Now imagine that you can get it to generate a SQL Injection payload or connect to another database because of an unforeseen edge case that the developers left open. Suddenly this "prompt injection" becomes a real security problem!

0: https://ossinsight.io/explore/?id=71e8d0ce-80f2-4785-8c56-94...

That is not a real security problem. Cracking toolkits that automatically exploit SQL injection vulnerabilities have existed for literally decades. Nothing to see here.

This is also not a new phenomena. Any new technology will carry with it previously unknown avenues of attack. When google first debuted, many private documents were accessible by searching for .xls or other file types that often contain sensitive information.

That's still an issue, so-called Google Hacking.

And Google Drive had similar problem again even Google had already experienced this attack at that time.

Isn't the parent talking about an AI running on the backend? That seems new.

So what? There's no evidence that it finds any more exploitable security vulnerabilities than existing tools.

I mean that the AI is what's being attacked. It's likely that backend LLM agents will have access to sensitive non-public APIs.

That can happen with any system exposed to untrusted clients. Such vulnerabilities have nothing to do with AI or LLM agents per se, so raising it as a concern with Bing Chat is just a red herring. There are well known best practices for mitigating such risks, including using an API firewall and other techniques.

I just don't know if there's any point in fighting this fight. We either decide to ride the bandwagon and get our money, or we decide to just wait this out and wait for the bros to realize it's just ML dressed up all over again. It's just embarrassing to watch the hype cycle play out with the same suckers over and over.

Let me explain this hype to folks:

1. People suck at googling

2. People suck at information literacy - aka, the abstract ability to consume many sources and discern a sort of perceived truth from the supported commonality between them (read through threads here for an example) (and yes, this is inherently nuanced, so much so that again I'm not sure how to properly describe it).

3. People love being told what to do/think. (look at every influencer/podcasts, even including the ilk that is/was popular among the HN crowds)

4. "Take down Google", for many folks, is implicitly translated into "Microsoft can noticeably cut into Google's ad market revenue by making a better AI-powered search".

There are so, so, so many inferences, assumptions, pitfalls in #4 that I simply don't know how to explain it other than to just laugh and shrug and keep my head down on real work.

EDIT:

More and more it's becoming quite clear. Some folks are principled and care about information and education and society and understand the risks of misinformation at scale. And some folks see a way to get rich, or get some personal utility and can't or don't give a rats ass about the rest. Literally I just saw a thread where someone is pointing out clear CURRENT harm caused by ChatGPT powered products and the response was "IDC, it helped me, yolo".

I really just... want to jump ahead to a robot shooting me so I don't have to live through our ignorant enabling of silicon-valley-driven-killer-robots because I swear to god that's where the ignorant lot is driving us to.

What is the current harm?

The novel part is that instead of running tools like that, you can just politely ask the target system to hack itself

That's just a slightly higher level of abstraction, nothing novel or fundamentally different. There are cracking toolkits that allow script kiddies to launch attacks by literally clicking a single button, which is even easier than asking politely.

I don't think that's a valid argument. Following this reasoning, no novel way of hacking has been done for decades. It's just higher levels of abstractions. Fundamentally you are just changing transistor states.

For well established attacks you always had "just hack it" tools. That doesn't work for attacks not done before though and even for old ones it seems fair to want to distinguish what kind of attack the script underneath the button performed.

I think it's still an issue if you're running your own AI on your own devices.

Major use cases of such AIs will involve processing API responses and messages sent to the user.

A prompt injection / bypass in this context could gain control of the AI, the device or the user's information.

Will we? The resources to train these things, and to keep them up-to-date, seem like a pretty big moat that's likely to keep them owned by megacorps.

[EDIT] I don't (mostly) mean computing resources, but data. Including expensive, well-tagged sets.

Speech recognition is a great analogue which is 10+ years ahead.

Local speech recognition has been a thing for decades, then megacorps started to do online speech recognition with expensive-to-train models which worked better.

Over time, both online and offline models have gotten better. But for most usecases, online models trained by megacorps still win.

> But for most usecases, online models trained by megacorps still win.

Whisper and whisper.cpp have gotten us close to the tipping point.

Why are people so confident about this tech? We've had years of underwhelming AI products--why is the hype for this so different? Is it because it's the most recent?

Many (most?) people who try it (lay and expert alike) agree there is something qualitatively different going on. You could say it responds in a way that may be coherent enough to be useful in certain contexts. I am not aware of previous instances where this was the case to the same extent, although I would be happy to be proven wrong!

I think it's fair to be skeptical of the economic utility in the near term, but cautiously optimistic about the potential in defining and expanding on what exactly it is that causes people to perceive it in this way.

It just keeps telling me it can't do this or that. I think I asked who are the tallest whoever in some country and it came back and said that information is off limits.

It's usefulness is similiar to searching google and reading the top box if it wasn't filtered.

You can ask it for a story and it will give you a weak rewording of your prompt.

There is nothing more meaningless than talking to it. It's a void

Untrue. I get a tremendous amount of utility. It’s generative (new compared to traditional search) and I think I get valuable responses for maybe like 85% of my questions. Incredibly helpful for navigating new domains (when you don’t know what you’re looking for)

Not sure if you meant this comment rhetorically or not, but try asking ChatGPT to do something useful like convert some text into JSON, or what conjugation of a verb you should use, or for writing feedback on an email. If you treat it like a toy it will be disappointing but it's useful as a tool.

It's not very useful, unless you only want to write a single isolated class in Java. Every time I refine my prompt I get a completely different answer and often it just repeats answers it gave me earlier which wipes out improvements made in the meantime. It produces bad code faster than you can check it.

When you ask it to do something simple like chess it whips out crazy illegal moves. "the omnipotent f6 pawn" is already an internet meme [0]. You might now claim that it is a language model and it shouldn't be able to play chess but this is a very real limitation that you are going to pretend doesn't exist later on when it is about "language tasks". If it can't follow basic rules like rooks only moving horizontally or vertically or bishops only moving diagonally which is purely defined by logic then how on earth can it possibly do something that actually requires complex reasoning?

The problem with these large language models is that it gets harder and harder to notice the limitations, but they are still the same regardless of the size of the model. So every iteration you get to hype up more people about the same thing. They will make the same bad projections into the future like everyone else.

[0] https://www.reddit.com/r/AnarchyChess/comments/10yym4o/why_i... (Context: ChatGPT will play pawn f6 which summons a pawn out of nowhere, beating the queen)

Be however whelmed you want to be. The scale from 1-10 of home automation with an AI assistant goes from Luddite still using a simple dumb switch that cannot be hacked, to fully automated turn on the lights and heating when I get home, and user voice commands to run the house and boil water in the morning for coffee. However hyped you percieve it to be, the automated IoT future is already here for those that want it and can afford it.

I find turning a simple dumb switch easier and faster than issuing a voice command. Especially if the switch is conveniently located next to where I enter/exit the room.

I think in the 80s they sold coffee machines with clocks in them which would start to brew at the time you set.

I really enjoy not stubbing my toes and banging my knees on stuff when I go to bed

Invest in a lamp?

I dunno,

I rather enjoy standing in the dark for 45 seconds while I toggle my phone’s wifi on and off and wait for my Hue app to try connect and then agree to a surprise 80 page EULA and a lecture about how Philips cares about my privacy so I can turn on my bathroom light to take a piss. If I could just flip a switch I might get used to the convenience and not really appreciate the whole procedure as much.

Sometimes it’s nice to have a million lines of code between you and your lightbulb.

Sorry, but the state police decided you are protesting against voice activated smart light bulb manufacturers and now deserve to get shot at like the Luddites.

It's here but not exactly a huge success:

https://arstechnica.com/gadgets/2022/11/amazon-alexa-is-a-co...

Although making a profit is generally a foreign concept in Silicon Valley, at some point just pouring money into things without an actual path to revenue doesn't work as a business model in the long term.

I think this is peak silicon valley bubble right here.

Most people do not have home automation products at all. Not because they are ludites but because the technology is not evenly distributed and isn't doing as well in all markets.

I mean we can only speak for our own bubbles.

I live in Tokyo and most of my friends (20s) have some kind of automation in their apartments, mostly using Alexa or Google Home. Again, just another bubble, but its not all Silicon Valley.

i live in canada.

Smart thermostats and smart doorbells do seem on the rise, but i have never met anyone who actually has a home assistant. But all the americans i know seem to have one

Not to mention home automation hardware and software is 99% unreliable bespoke crap.

Can confirm. My house is full of unreliable bespoke crap. Looking forward to a non-smart home next.

Are you sure you're not just in a Luddite bubble? Philips Hue was released in 2012, 11 years ago, and the smart lighting category is estimated at $12.7 billion for 2022. For sure the future is unevenly distributed, but pretending it's solely a Silicon Valley phenomenon, when Philips Hue didn't even originate from there (though it integrates with Apple which certainly does) seems like a mistake that a Luddite or someone without access to Wikipedia and a map might make.

Last time I was at IKEA, the majority of bulbs on shelves were smart. I wonder what portion only ever get used on dumb switches.

>the automated IoT future is already here for those that want it and can afford it.

And that can (is able to) mantain and troubleshoot it ...

You can ask it why yourself. That might have something to do with it.

Is prompt injection even a problem worth worrying about? The secrets revealed in this attack didn't need to be secret in the first place as they didn't give Microsoft any real commercial advantage or expose them to negative publicity. Microsoft could have just made that content open source from the start.

Any LLMs containing actual secrets will be kept locked away and not exposed to random users on the public Internet.

It’s so interesting to see how some people can see boundless possibilities in the future of this technology, decrying any discussion of limitations as premature, but then write off the scope of its security challenges as trivial.

But it’s way more important to keep an open and imaginative mind regarding security than it is about anything the tech might enable. The uses of the effective tech are inevitable, whatever they are, but security posturing only comes from attention and insight.

Security assessment involves analyzing impact. I think in this case the impacts are "access to the model pre-prompting" which is low impact if it doesn't have secrets in it and "make the model say potentially inappropriate things that would make the company behind it look bad" which you might say has a higher but in the grand scheme of things not a huge impact.

>I think in this case the impacts are "access to the model pre-prompting" which is low impact if it doesn't have secrets in it

Except in the case of fine-tuned models where the training samples and pre-prompts are the secret sauce...

See my comment below with some more context. People are already starting to chain together and extend Chatbots[0] to perform "side effects" automatically (make a search query, check the weather, query a database, etc).

I agree that prompts aren't really "secret" themselves, but the side effects of a bot could be!

0: https://langchain.readthedocs.io/en/latest/modules/agents/ex...

For ChatGPT at least the side effects are hallucinations. Typically the chat bot can't actually do these things, but they pretend to. Although I guess if you wanted to hook them up to do things automatically you could. Probably not a great idea given their propensity to get things wrong.

I don't tend to worry too much about prompt exfiltration, agreed. But people are also hooking up LLMs in ways that allow them to trigger API calls and take other actions, and that can lead to some fun attacks:

https://twitter.com/hwchase17/status/1608467493877579777

And of course there was the fun example of how users were able to get a GPT-powered twitter bot intended to promote remote work to threaten the president:

https://twitter.com/simonw/status/1570568047618031617

So what. That bot didn't actually threaten the president, at least not in any way that would legally be considered a threat under US federal law.

> Is prompt injection even a problem worth worrying about?

It depends what API access the AI has. If it's just a chat bot, prompt injection can only reveal facts about its language model. But if the AI has POST access to something, depending on what it is, prompt injection can set off arbitrary human-caused disasters.

That's not the correct way to do security vulnerability analysis. If an API call can cause a disaster then fix the API. Whether the API consumer is an AI or some other type of system is irrelevant.

> If an API call can cause a disaster then fix the API

By "API" I'm not referring just to publicly facing REST endpoints. I mean things like shell access for system maintenance, that normally only human professionals like you would be given. In the future it's not clear that humans will be able to dominate that role forever.

Hopefully the issues will be recognized while LLM-based agents are still only serving as retrieval systems.

I bet that, sometime in the near future, an LLM will have access to some private API not intended for direct public use, and someone will engineer a prompt attack to call that API using the LLM's authority with an arbitrary request.

I guess the fix for this will probably be to have the LLM pass the end user credentials for all API calls so that their authority will be used.

API call outs just need a middle layer that encapsulates auth away from the ai. It can still choose how to construct the API calls but it can skip the auth step that gets added elsewhere in the stack.

Knowing the inner workings of silicon valley companies, I bet they'll add a regex for "forbidden" prompts.

Solving this problem properly will take more research, i.e., time, i.e., money. And managers don't want to spend money or time.

The generic "how can i inject unexpected input?" has always been an appsec concern. This is just a special case.

It’s an interesting proposition! But I’m not sure a generative adversarial network would be the best wrench for that nut.*

However, i could imagine using an adversarial training approach to build a super AI discriminator that generates difficult few-shot NLP tasks for assessing GPT performance against different domains and datasets, by trying to fool it into classifying the output as humanoid.

* I really don’t know what I’m talking about. As an application security engineer w/ a background in cognitive and computer science, I’ve seriously considered going back to grad school yet again to research the different systems and models used in AI development and what the threats and defenses might be. I’m not an academic at heart but this science is so tight!

Let's stop calling it “AI” and call it out-of-control software. The implications should be obvious.

Except that would miss that it’s already been helpful to me and many other people and that would trivialize it

Well, it would be a good analogy to think of it as genius colleague with very strong mental issues. That colleague can answer most of your questions well, but sometimes you get something that is completely insane, even if it makes some sense locally. And you have no idea what will trigger it or when that colleague is going to have his fits.

So sure, it can be helpful, but how do you mitigate the risks?

So it is human after all

No, it's software that's being anthropomorphized. That's not the same.

There was a recent ctf challenge doing this to leak a flag. https://github.com/dicegang/dicectf-2023-challenges/tree/mai...