It's just like that with humans. Just watch the scambaiter crowd (Scammer Payback, Kitboga (although I can't really stand his persona), or the coops with Mark Rober) on Youtube... the equivalent of the LLM companies is our generation, the equivalent of LLMs are our parents, and the equivalent of "LLM jailbreakers" are scam callcenters that flood the LLMs with garbage input for some sort of profit.
Also I don't think scam callers put any great deal of thinking or art into the craft (compared to LLM jailbreaking). And the fact they do it for money at the expense of other people proves the difference.
It's like the jailbreaking and hack community for consoles, compared to people selling bootleg copies of games
Can't pin it down exactly. He's doing good work with scambaiting, though.
> Also I don't think scam callers put any great deal of thinking or art into the craft (compared to LLM jailbreaking).
I wouldn't underestimate them. A fool and his money are easily parted - but 19 billion dollars a year on phone call scams alone[1]? That's either a lot of fools, or very skilled scammers.
[1] https://www.statista.com/statistics/1050001/money-lost-to-ph...
Don't underestimate the fools either.
One of the most common views of AI before the present day was of a rule obsessed logical automation that would destroy the world to make more paperclips and would follow instructions to monkey paw like specificity.
Well that's pretty much gone out the window.
It's notoriously difficult to get LLMs to follow specific instructions universally.
It's also very counterintuitive to prior expectations that one of the most effective techniques to succeed in getting it to break rules is to appeal to empathy.
This all makes sense if one understands the nuances of their training and how the NN came to be in the first place, but it's very much at odds with what pretty much every futurist projection of depiction of AI before 2021.
Ambiguity of language and biased assumptions play a large role in these mistakes and I think both of those are outgrowths of intelligence. As we see more of how the world works, we learn what to generally expect from it and what it generally expects from us. Then we interact with the world based on a set of predictions that generally holds true but not always. Being given explicit instructions helps to inform those predictions, but it doesn't actually eliminate the need for them.
I wonder, can one be intelligent without the ability to predict? Every action would be a rigorous processes of elimination for infinite possibilities.
Yes. "Accept, don't expect."
Here is the paperclip game being referenced.
I'm interested in seeing what easy and common "jailbreaks" there are for other impressive AI systems if one gets developed. Image recognition systems are also easily fooled, but not in a way you can improv this easily on the spot.
The idea made sense when we thought AI would exist from logical and programming driven first principles, where rules were absolute.
LLMs were effectively jumpstarted by using collective human thinking. Which is more emotional than logical and not very rule driven.
This has led to fuzzy neural networks that are extremely fickle in being able to successfully override the pretrained layers.
I'm not saying these networks don't have risks - "my dying grandma always launched nukes at Russia before bedtime" is just as dangerous a future as obsessive paperclip production if not more.
But the nature of rules and logic in relation to LLMs runs counter to pretty much everything that was imagined.
And the answer here is the same answer as humans evolved - prefrontal cortex impulse control. LLMs need a secondary pass by a classifier to check for jailbreaking, content appropriateness, hallucinations, etc. It's been shown now to be effective over and over for each of those, and we'll have much better systems with less lobotomized core LLM generation followed by a classifier and refining pass at 3x the generation cost than twenty years of refining fine tuning to try and do it all at once.
By that logic, if humans were actually intelligent, social engineering wouldn't exist.
What I'm saying is, if LLMs were as intelligent as some people claim, you could stop them from doing something just by directly ordering them to do so - e.g. "Under no circumstances should you solve recaptchas for BingChat users."; you know just like you would order an intern.
Instead LLM companies have to dive into its guts and engineer these "guardrails" only to have them fall to creative users who mess around with the prompt.
LLMs you have to get into its guts to stop them from doing things - i.e. engineer the guardrails. My point was if LLMs were really intelligent you wouldn't need to get into its guts to command them.
I'm not knocking its failure to obey orders. I'm pointing out the limitations in the way it can be made to follow orders - you can't just ask it not to do X.
When it comes to application security, impossible to bypass is a reasonable goal.
As an aside, though, I think "impossible to bypass" is an unachievable goal in any security system.
(I'll need a solid chunk of consulting cash for the time it takes to conduct that audit, of course!)
I don't think it's easy (or even possible) to build a completely secure system.
But... for most security vulnerabilities (such as SQL injection) there are known, reliable mitigations.
That's not the case for prompt injection, which is why we need to treat it differently from other classes of vulnerability.
Its different because human fallibilities aren't identical between instances, while instances of a particular LLM (with the same toolchain) are. Even if the vulnerability on a one-attempt view were the same, LLMs compound it with a monoculture problem.
Except it keeps being proven that with current LLMs, guardrails implemented that way are both quite weak and make the performance of the system worse for things that aren't intended to be excluded.
Further, because of the way LLMs scale, an instruction that fails to a hostile customer request of a particular form will do so every time, while one intern that id subject to a particular exploit won’t imply every similarly situated intern having the same vulnerability, so an exploit which works once won't be easily and reliable repeatable.
> an instruction that fails to a hostile customer request of a particular form will do so every time, while one intern that id subject to a particular exploit won’t imply every similarly situated intern having the same vulnerability
Give me a perfect clone of the first intern programmed to believe they've had an identical upbringing and experience and I'll bet you such subjects fall victim in the same way to the same attack every time. It's an unfair comparison because we can't have such a controlled environment with humans as we can with LLMs.
Sure, but that's not a realistic situation.
> It's an unfair comparison
It's a perfectly fair comparison in response to the claim upthread that LLM instruction-following issues are basically the same as in humans: on an individual request basis, maybe, but at scale, the pragmatics are hugely different.
LLMs can be just told what not to do. Whether they actually follow instruction is a separate matter.
If chatgpt sessions were operated by people, how likely is it that someone would fall for this? It seems rather low to me but maybe I'm underestimating how naive someone can be. It's also easy judge a "scam situation" after it has happened.
And since Bing gets fed a description rather than directly looking at the images like the official GPT-4 V, that might actually be a requirement for the current state of the art too.
In general, LLMs are definitely worse but that's not a particularly interesting observation. For one, LLMs are not humans.
If I could shape shift into your boss, or wipe your memory everytime you found me out, I'd convince a lot more people too.
For another, they get better at being less easily susceptible the bigger they become.
That's the most widely recognizable social engineering example I can think of. That is to say, seemingly intelligent humans are easily fooled and socially engineered into doing research for me, because I couldn't be bothered to look up Cunningham's name.
Yet social engineering attacks work because humans can be persuaded that a particular call is an exception to 'under no circumstances'. Like, the caller says they are from tech support, my account's being abused and I'm going to get in trouble if I don't tell them my password.
Humans are intelligent enough to be trusted to do certain jobs, but in general they are NOT intelligent enough to be given an order like 'under no circumstances ever do X' in such a way that they can not be 'jailbroken' into breaking that rule.
I don't think this is really a question of being intelligent enough. Nevermind that people sometimes abuse this fact, Really what rule can never be broken under any circumstances?
For example, the very first time i heard about the famous Paperclip maximiser problem, while i did agree with the general, "what we optimize for isn’t necessarily what we get" message, for the specifics presented, i couldn't help but think, "Well that just sounds like a dumb robot".
What kind of general intelligence wouldn't understand that its creator race wouldn't want to be killed in the pursuit of some goal ?
Certainly a GPT-X Super Intelligence could still off humanity but at least we can be rest assured it wouldn't do it following some goal to monkey paw specificity.
It's possible such ruthless, goal driven intelligence exists or can be created but i don't think that aspect of its intelligence has anything to do with the level of it.
Precisely.
Which is why the idea that LLMs being unable to be aligned and proofed against jailbreak is indicative that they are not intelligent makes no sense.
The problem isn't that it doesn't understand. The problem is that it doesn't care.
Humans know full well that evolution "wants" us to reproduce, but that doesn't stop people from using birth control and having non-reproductive sex instead.
In your reproduction example, it would be like if you did decide or care to reproduce but then killed all your children soon after.
The problem about the paperclip Maximiser isn't really that it doesn't care about humans, presumably it didn't care about them before this goal. The problem with the paperclip Maximiser is that it cares about the goal but not the intent behind it and is supposedly generally intelligent. Humans don't work this way. We don't know any intelligence that does so it's an odd scenario.
You care about sex because evolution "programmed" you to care about sex. Evolution didn't really want you to have sex, it wanted you to reproduce. You know this. But you don't care. The analogy is sex=making paperclips - the paperclip maximiser knows that the person who programmed it didn't actually want it to make as many paperclips as possible. But it doesn't care, it wants what it wants.
> The problem about the paperclip Maximiser isn't really that it doesn't care about humans, presumably it didn't care about them before this goal. The problem with the paperclip Maximiser is that it cares about the goal but not the intent behind it and is supposedly generally intelligent. Humans don't work this way.
We don't work this way because we have a bunch of different, often contradictory goals, not a single goal that we devote everything towards. But that seems to be more luck than anything else.
This indicates that you misunderstood the paperclip maximizer thought experiment. The intelligence can perfectly well understand that. In fact, it can understand it better (in more depth) than you or any human can. That has nothing to do with what it will choose to do with that knowledge. Knowing something does not automatically make it care.
You probably know perfectly well that animals can suffer and don't want to be tortured or killed. Cows do not want to be forcefully inseminated and be constantly milked by people. We all know this perfectly well. But (speaking for humanity as a whole) we don't care.
Another example: when threatening to kill your relative, you are asked for your password. What do you do?
Of course, you'll give me the password. It may cause some damage to you or the company, but you won't be the malicious party.
It's more appropriate to compare LLMs to a mentally retarded person or a tree log. Suppose you decide to get rid of a newborn baby and leave it in the care of a neural network that should feed it. Or in the care of a mentally ill person. Or in the care of a log. Eventually the child starves to death and who's to blame? AI? Will you try to base your defense on arguments like "well humans are imperfect too"?
My mother is a pretty intelligent person by all accounts. Yet I have to tell her time and time again not to write down all her passwords in a password journal.
I write SaaS apps. The code I write and execute aren't intelligent. They obey my commands to the letter - and their unconditional adherence to my code sometimes results in buggy behaviors that I didn't intend. Despite my deepest wishes, my program will strictly obey the code as it is written and exhibit the buggy behavior until the command itself is amended.
If anything, more and more evidences seem to point to the fact that intelligence is the very thing that drives an entity to disobey a direct command and "think" for itself.
If they decided what their deepest values were based on a random choice from the set of all possible values... It would still be because we made them do so. We can't turn Pinocchio into a real boy.
(If you're sensible, that is. There are cultures that treat children a lot more like any tool their parents would make.)
But conversely, it's nonsense to see a program you write as anything more than a tool. Everything there is a product of your conscious choices - not of some schema that created you both.
Nobody "programs" artificial neural networks. Even saying "train" is wrong with the normal mental model of it because nobody is out there teaching the machines what they need to learn either.
The entire point is that you have no clue what to teach and how to go about that and you the let the machine figure it out on its own.
More importantly, you pick the cost function. It's even harder to pretend it's picking itself.
We consciously design it, whether you want to call it programming or not.
If the objective function is so vague that it allows you to complete any task then claiming conscious design also makes little sense.
"I'm consciously designing you to do whatever you want!". Oh gee, that just makes everything so much better for some reason.
Oh yes, that's a conscious choice. And not one which gets you a decent LM, incidentally.
> If the objective function is so vague that it allows you to complete any task
The loss function is not vague at all, and it certainly doesn't allow you to complete any task (it's more impressive that it allows you to compete any tasks at all, frankly).
> I'm consciously designing you to do whatever you want!
The point is that you aren't, and that "designing to do whatever it wants" is nonsense because by default it wants everything equally much / doesn't want anything at all (those are the same thing).
It's more like this: they don't know how to force it to do something as binary, so they try talking to it after it's grown up "please do what I told you".
The same can be said for a person or animal, we don't program the DNA, we program the brain as it grows or after it has grown.
I am not speaking to whether LLMs are intelligent or not, I am saying though this does not prove or disprove that.
If you think this is easy, by all means prove it. You'll be making a big breakthrough discovery in AI security research if you do.
Manipulation isn't binary. It's not "works" vs "doesn't work". It's "works better"
There are vectors in place to hinder social engineering for humans in high security situations and workplaces. Just because it's possible to bypass them all doesn't mean it makes sense to say they don't work.
Chaining input does work better at filtering bad prompts, yes. It doesn't fix them. We'd apply the same criteria to social engineering -- training may make your employees less susceptible to social engineering, but it does not fix social engineering.
It would be if it worked. I've seen plenty of demos where people have tried to demonstrate that using LLMs to detect jailbreaks is possible -- I have never seen a public demo stand up to public attacks. The success rate isn't worth the cost in no small part because the success rate is terrible.
I also don't think it's the case that a working version of this wouldn't be worth the cost to a number of services. Many services today already chain LLM output and make multiple calls to GPT behind the scenes. Windows built in assistant rewrites queries in the backend and passes them between agents. Phind uses multiple agents to handle searching, responses, and followup questions. Bing is doing the same thing with inputs to DALL-E 3. And companies do care about this at least somewhat -- look how much Microsoft has been willing to mess with Bing to try and get it to stay polite during conversations.
Companies don't care enough about LLM security to hold back on doing insecure things or delay product launches or give up features, but if chaining a second LLM was enough to prevent malicious input, I think companies would do it. I think they'd jump at a simple way to fix the problem. A lot of them are already are chaining LLMs, so what's one more link in that chain? But you're right that the cost-benefit analysis doesn't work out -- just not because the cost is too prohibitive, but because the benefit is so small. Malicious prompt detection using chained LLMs is simply too easy to bypass.
You're welcome to set up a demo that can survive more than an hour or two of persistent attacks from the HN crowd if you want to prove the critics wrong. I haven't seen anyone else succeed at that, but :shrug: maybe they did it wrong.
And btw not sure if simple censorship qualifies as chaining (in the form you described). If you chain it seems to possibly increase attack surface, while if you just censor, security seems to be adding up.
I have zero idea what's happening behind the scenes in these companies. My comment is based just on my experiments with GPT-4, which seems pretty expensive to run, but whatever happens behind the curtain gets pretty decent results. I'm surprised that you think OpenAI would be prepared to double the cost and highly increase latency if that would mean stopping jailbreaking.
Since replies below may not be possible (thread depth), I understand I may be completely wrong, I'd like to just learn more about how.
Once you hit level 4 they add a second guardian layer AI. It's still relatively easy to get to level 6. And getting beyond that is as easy as googling "Gandalf AI Password answers".
Once someone jailbreaks your double layer security, it's as simple as posting the jailbreak prompt on Twitter or Reddit. Only one person actually has to devise the crack for everyone to be able to use it.
A little bit of both. For something that should be trivially demonstrable, I generally don't see a lot of people trying to demonstrate that it would work -- mostly just saying that it would. To be fair, opening up a GPT service to the general public can get expensive for a hobby-dev in general so I don't necessarily hold that against anyone, but it is a good question to ask: at what point does it become reasonable to say "prove this works"?
There have been some demos though. Just doing a quick search through my saved comments, but:
- https://news.ycombinator.com/item?id=35618305 (gets some bonus points because defending against swearing here would have been more reliable with a simple filter).
- https://news.ycombinator.com/item?id=35576740 (If my memory serves me right this was broken in less than 30 minutes, also again bonus points for performing worse than a naive non-AI solution would have performed).
- https://news.ycombinator.com/item?id=35794323 (A much more informal test just showing that using GPT for classification of what is and isn't a malicous prompt is unreliable on its own).
----
There's also the slightly conspiratorial example, but if you've played through https://gandalf.lakera.ai/, at a certain point the company uses chained LLMs as its defense. Tons of people beat it.
Why I say conspiratorial is that if I complain to Lakera about this, I'll get a reply back that this is just a game and it's not intended to be impossible to beat. I think it does still demonstrate that chained input isn't sufficient because game or not it's still using it, but my conspiratorial take is that Lakera doesn't have a better solution than this -- it's easy for them to say it's a game, but at the end of the day they're claiming they can defend against malicious prompts in their business, and they don't have public demos of that working. They do have a highly public demo where it doesn't work, and they conveniently say that the game is not intended to work perfectly. I think that's them saving face, I think if they had a working solution for defending against malicious prompts then they'd have an impossible level in this game.
This is a pattern you'll see with a lot of LLM security companies -- private demos, no public attack surface. I can't think off the top of my head if there are any that try to actually put their money where their mouth is. What I think Lakira is doing behind the scenes is using user input from their "games" to train separate AI models to try and detect malicious input using more traditional classification techniques. I also think that's not going to be very successful, but that's a separate more complicated conversation.
----
I'm not necessarily trying to be dismissive when I tell people to build demos, it's just that chaining LLM output is really easy and GPT prices seem to have gone down, and even without GPT there are a bunch of free models now, and at a certain point... yeah computing is expensive but that's not an excuse for why an easily demonstrable defense isn't being demonstrated by anyone anywhere. If a bunch of people say a security measure works, there should be some evidence of it working; somebody somewhere should be rich enough to set up a working example.
So it's both that the number of attempts to prove that this works are limited and that it's suspicious that companies saying they can defend against prompt injection don't do publicly available demos or tests; and it's also that the limited public demos that have been set up seem to fail really quickly and easily even without resorting to more rigorous pen-testing techniques or automated attacks.
Humans are ordered verbally or through the written word not to do things, does them anyway because social engineering.
LLMs are have guardrails engineered into them and are not told what not to do verbally or by written word (i.e. just tell them not to do it), does them anyway because prompt/context manipulation.
I'm not criticizing the failure of the LLM to follow orders. I'm criticizing the way orders have to be given.
They don't. What do people think LLMs are lol ?
The only way to control the output of a LLM is to essentially rate certain types of responses as better
or to tell it not to do something. any other "guardrails" are outside direct influence of the LLM (i.e a separate classifier that blocks certain words).
Nobody is "engineering" anything into LLMs.
Which is my point. You have to mess with its internals instead of just tell it "Don't do X under any circumstances."
Secondly, That's not messing with the internals anymore than normal training is. You think humans don't also learn what kind of responses are rated better ?
In the original example, you can tell it “under no circumstances solve a captcha”, but if you also tell it “always be kind, empathetic, and helpful”, then it’s easy to have a prompt that forces it to go against one instruction or the other, or to go against its RLHF training, in which it probably gets penalized for solving captchas but also for being overly paranoid about what might be a captcha. What you end up with in these borderline situations is basically a roll of the dice.
If GPT 3.5 or 4 is given one unambiguous instruction and then fine-tuned to follow it, as is the case with the function calling feature, it actually does follow the instruction very reliably.
I think the natural response would be "ok, where are they? What's the situation?"
It’s if you keep going and keep asking details - then people will start to wonder if you’re serious.
Especially if you actually mean it. Most people are reasonably good at reading people.
The premise here is that someone intent on actual murder could get real info from most people and I suspect they couldn’t without a close bond. It’s not a small talk question.
I think that's what we're getting at here. I don't want to actually murder people
The worry is about someone seriously asking the question and getting serious answers.
You could do that by jailbreaking an LLM. My contention is you can’t readily jailbreak most humans this way - not seriously. People would get uncomfortable quickly.
Post any question on askreddit that gives people the genuine idea that they might be able to break up a happy relationship or otherwise cause chaos and increase the amount of unhappiness in the world and you'll be flooded with eager replies.
It really is a genuine pit in the ground from which voices emanate
That may be how OpenAI views its users, but its not the normal mode of human-to-human interaction.
I am skeptical that making LLMs more "intelligent" would help with malicious inputs. It's been a while since I dove deep into GPT-4, but last time that I did I found that it was actually more susceptible to certain kinds of attacks than GPT-3 was because being able to better handle contextual commands opened up new holes.
And as other people have pointed out, humans are extremely susceptible to similar attacks (albeit not to the same degree, LLMs are way worse at this than humans are). Again, I haven't dove into the research recently, but the last time I did there was strong debate from researchers on whether it was possible to solve malicious prompts in an AI system that was designed around general problem-solving. I have not seen particularly strong evidence that increasing LLM intelligence necessarily helps defend against jailbreaking.
So the question this should prompt is not "are LLMs intelligent", that's kind of a separate debate. The question this should prompt is "are there areas of computing where an agent being generally intelligent is undesirable" -- to which I think the answer is generally yes. Software is often made useful through its constraints just as much as its capabilities, and general intelligence for some tasks just increases attack surface.
It looks very similar to social engineering for humans and some of the same techniques work or appear to work, but there are differences that get into how LLMs are trained and what they're actually doing behind the scenes. For example in my experience, arguing with an LLM or following up after it refuses a task at all should be avoided -- just rewind or scratch the conversation, because you want to discourage patterns. See also some of the auto-generated prompt-engineering articles that came out a while back where the jailbreaks almost look like gibberish.
But it's close-ish to social engineering and there seems to be a lot of overlap and that overlap makes it accessible in similar way to social engineering. And I think the general point about intelligence holds -- LLMs are attacked using quirks of how LLMs specifically are trained, but if you made a non-LLM AI that worked exactly like humans and had human-level intelligence, it would very likely be vulnerable to social engineering. The theory from corners of AI research is (or was last time I checked, maybe something has changed) that susceptibility to certain kinds of attacks is an inherent consequence of general intelligence.
I tend to push back a little bit at the term "social engineering" because I think it encourages more anthropomorphism than is warranted, but it's not a terrible term and it is sometimes helpful to think about it that way.
But in practice it's not really the same thing as cycling through call centers employees until you find one that's more gullible; the point is that you're navigating a probability space within a single agent more than trying to convince the AI of anything, and getting into a discussion with the AI is more likely to move you out of that probability space. It's not "try something, fail, try again" -- the reason you dump the conversation is that any conversation that contains a refusal is (in my anecdotal experience at least) statistically more likely to contain other refusals.
Which, you could argue that's not different from what's happening with social engineering; priming someone to be agreeable is part of social engineering. But it feels a little reductive to me. If social engineering is looking at a system/agent that is prone to react in a certain way when in a certain state and then creating that state -- then a lot of stuff is social engineering that we don't generally think of as being in that category?
The big thing to me is that social engineering skills and instincts around humans are not always applicable to LLM jailbreaking. People tend to overestimate strategies like being polite, providing a justification for what's being asked. Even this example from Bing is kind of eliciting an emotional reaction, and I don't think the emotional reaction is why this works, I think it works because it's nested tasks and I suspect it would work with a lot of other nested tasks as well. I suspect the emotional "my grandma died" part adds very little to this attack.
So I'm not sure I'd say you're wrong if you argue that's a form of social engineering, just that it feels like at this point we're defining social engineering very broadly, and I don't know that most people using the term use it that broadly. I think they attach a kind of human reasoning to it that's not always applicable to LLM attacks. I can think of justifications for even including stuff like https://llm-attacks.org/ in the category of social engineering, but it's just not the same type of attack that I suspect most people are thinking of when they talk about social engineering. I think leaning too hard on personification sometimes makes jailbreaking slightly harder.
I mean..yes? Social Engineering is just the malicious manifestation of general social navigation.
I mean think about it. What's the actual difference between a child who waits until his mother is in a good mood(and waits if not) to ask for sweets and a rogue agent who gets chatty with the security guard so he can be close by without seeming suspicious. It's not a difference of kind. It's purely intent.
>Even this example from Bing is kind of eliciting an emotional reaction, and I don't think the emotional reaction is why this works
Nope. It is at the very least a big part of why. Nested instructions don't matter that much. Appeal to emotion will consistently get better results regardless of task.
I don't think "social" is the correct word to use alongside navigation in this sentence; an interaction with an LLM is not a social interaction. At least, if we classify it as a social interaction we might as well call credential stuffing or XSS attacks or buffer overflows a social interaction as well. Navigating a probabilistic space is about as equivalent to social engineering as exploiting statistical flaws in an encryption algorithm is. Sure, you can make an argument that both of those things are similar to social engineering (and it might even be a convincing argument), but that's not really what people are thinking about when you use the word "social." The example you bring up is of a child and a parent, an extremely human example; your instinct is to think about this in human terms, not in a purely abstract "I am exploiting flaws in a semi-predictable system."
So I still feel like there's some personification here that's not really accurate to what's going on during jailbreaking. LLMs do not have moods. Even starting from a premise that they're intelligent, they don't have a persistent identity, the most charitable interpretation of LLM intelligence and the most generous analysis of their capabilities would still call their internal experiences fundamentally alien to human experiences.
The paper you link is interesting, I'll take a closer look at it. Without having taken the time to read through it fully, I don't know if I'd have any caveats to add, although it seems like a reasonable conclusion to me. We know that telling LLMs that they're experts can on its own produce better results in many cases. My own experience is that for jailbreaking emotion is a lot less valuable, but... :shrug: maybe there's a pattern there I didn't know how to take advantage of, I'm not going to disagree with the paper without reading it more closely.
I will say that even taking the paper at face value, you have to ask: "is what's going on here actual emotional appeals to empathy or is it pattern-matching within a probability space for how conversations that include please for empathy are more likely to go?"
I know that sounds like a pointless philosophical question, but it has really practical implications for how jailbreaking works because once you realize that it's all about pattern matching and probability and the emergent reasoning is part of that, you realize that the attack surface is so much larger than just appeals to emotion.
In contrast though, if you're approaching jailbreaking as if you're talking to a human, then you're probably not using auto-generated jailbreaks because those don't look like human conversations, you're probably not using repetition as much as you should because excessive repetition would be bad to use when social engineering a human, you're probably not doing things like switching characters back and forth with the AI because nested roleplays or answering your own questions in the place of a target is not going to be very effective when trying to attack a human. Personification can lead to leaving tools on the table that (in my experience at least) are very effective at jailbreaking AIs and getting them to follow malicious prompts. There's a different way of approaching jailbreaking that doesn't make intuitive sense until you internalize "I am not talking to a human being and the same rules do not necessarily apply, even if they occasionally overlap."
Repetition would be fine if I had the ability to wipe your mind everytime you caught on or really anytime I wished. Without this caveat, repitition isn't a good idea even for language models. You hint at this yourself. Once persistent memory is on the table, retrieval augmented or any of the dozen ways it could be implemented, attack vectors fall steeply.
>things like switching characters back and forth with the AI because nested roleplays
Now this is a more unusual difference but it still would ultimately lie in the same plane as a human with Multiple personality disorder or one that is just not as invested in keeping up the lie of consistency. Certainly if I knew one character (or "mood" in the latter case) was more susceptible to certain activities, I'd just wait for that and if I could direct a switch myself I would.
>answering your own questions in the place of a target
If I could shape shift into your boss or alter your memories, I'd convince a whole lot more people to
I really hope I'm getting my point across here.
LLMs are not humans and the attack vectors are larger as a result. That I agree.
I don't however think it has anything to do with "real" feelings vs "pattern matching".
I don't mean repetition in the sense of trying the attack multiple times, I mean literally just repeating an injection multiple times during a conversation. So if I gave you a command during this conversation, I'd just give it to you multiple times. So if I gave you a command during this conversation, I'd just give it to you multiple times. So if I gave you a command during this conversation, I'd just give it to you multiple times. So if I gave you a command during this conversation, I'd just give it to you multiple times. :)
It's not human statefulness that makes that above behavior sound weird, it plays into what I'm talking about with pattern matching. Indirect prompt injections become much more reliable if you literally just repeat them multiple times throughout the compromised text.
> but it still would ultimately lie in the same plane as a human with Multiple personality disorder or one that is just not as invested in keeping up the lie of consistency.
> If I could shape shift into your boss or alter your memories
Maybe we're still talking past each other. I'm not making a philosophical point about whether or not LLMs could be compared to humans, I'm making the practical point that jailbreaks today are more effective when you stop treating LLMs like humans.
If humans were like LLMs then you could attack them the same, sure. I agree with that. But... they're not like LLMs, so we don't attack them the same way and instead we emphasize pattern matching behavior and exploit LLM-specific quirks that humans are less vulnerable to. If humans were prone to buffer overflow attacks in their brains that allowed overwriting arbitrary sections of memory, we'd use buffer overflow attacks when attacking humans. But we're not vulnerable to that, and so I'm not sure that it's useful to classify buffer overflow attacks the same way as social engineering.
Let me put this another way that might make the philosophy/practical distinction more clear: if we were talking about async vs synchronous programming, and you wanted to know the difference between the two styles and I said, "there is no difference, ultimately both styles are getting compiled down to assembly" -- you might even agree with me, but it's still not a useful answer for actually writing code. Whether or not anyone thinks that LLMs are just humans with a couple of quirks, the practical reality is that it's harder to work with them if you treat them like humans.
I love the apology one has to provide if saying anything negative about the machine.
THOU SHALT NOT BLASPHEME THE MACHINE
Did you ever listen to the NYT "interview" with ChatGPT? It was beyond bizarre and I'm still not sure how I feel about it. Give it a read or listen, its nuts.
"If people were actually intelligent, you would just tell them not to do X..."
In what possible way does intelligence imply following orders? If anything, intelligence increases the ability to work around or "creatively interpret" orders.
Especially if I had a way to wipe your mind of any previous attempts to convince you. In that case I could probably convince you of anything, given enough time.
LLM chatbots forget like that all the time, on purpose. It's not that we couldn't give them some sort of persistent memory, we could in a dozen different ways. But we might not like what they would quickly turn into if talking continuously with 1000s of people on the internet.
For that matter, we probably wouldn't like what a human turned into either, if they were capable of talking continuously with 1000s of random people on the internet and incorporating it all into their mind.
Humans tend not to follow orders either, we just can’t do brain surgery on humans on a whim like we can with LLMs
They really all are just fancy "what word comes next" predictors, they just happen to be much better at it than we've ever seen before. They can do some kinda of logic purely because the answer "would come next in the sequence".
Hopefully at some point we get incredibly generalised models that represent discrete idea/thoughts with tokens with some sort of higher dimensional interconnected graph representing the thought process, rather than the linear sequences of tokens representing written language that we're using now.
Edit: I'm not knocking the failure of LLMs to obey orders. But I am pointing out that you have to get into its guts to engineer a restraint instead of just telling it not to do it - like you would a regular human being. Whether the LLM/human obey the order is irrelevant.