News for hackers. Stuff that matters.
Re:ChatGPT's image generator can be manipulated to produce v... by anuramat on Wednesday June 17, 2026 @09:27PM (#48579388)
> issues like prompt injection are unfixable

how is it unfixable? do you mean "there's always a positive chance"?

Re:ChatGPT's image generator can be manipulated to produce v... by dijksterhuis on Wednesday June 17, 2026 @09:35PM (#48579456)
normal

    y = f(x)
prompt injection / adversarial example (same thing really)

    bad_y = f(x+badness)
tweak badness enough you will get bad outputs. no matter the defences.

the only ways to fully “fix” it ie to make prompt injection never possible

1. don’t use ai

2. know the entire input space, output space and the mapping between them. but then we’re not doing machine learning anymore, see 1.

otherwise we’re left with mitigations. and mitigations are always a cat and mouse game with defenders (blue team) catching up. its never “fixed”. the latest thing just gets “patched”.

Re:ChatGPT's image generator can be manipulated to produce v... by anuramat on Wednesday June 17, 2026 @10:44PM (#48580052)
> tweak badness enough

assuming you get to do gradient descent AND the context is fixed+known AND you have unlimited compute? sure; is it a realistic setup?

> the only way to fix ...

the exact same argument applies to any (sufficiently complex) piece of software, with exactly the same conclusion

also technically I'd argue that we do know the input/output space (set of all token strings of length <= N/token), and know the mapping (the model is a ~pure function in terms of the api, which is about as good of a representation as it gets for a non-invertible mapping); at least it's much closer than with something like linux

Re:ChatGPT's image generator can be manipulated to produce v... by solid_fuel on Wednesday June 17, 2026 @11:27PM (#48580407)
> assuming you get to do gradient descent AND the context is fixed+known AND you have unlimited compute? sure; is it a realistic setup?

Clearly nothing so complicated is required, given the prompt in the very article you are commenting on.

> the exact same argument applies to any (sufficiently complex) piece of software, with exactly the same conclusion

Yeah and the halting problem is hard too, but there's levels to this shit.

> also technically I'd argue that we do know the input/output space (set of all token strings of length <= N/token), and know the mapping (the model is a ~pure function in terms of the api, which is about as good of a representation as it gets for a non-invertible mapping); at least it's much closer than with something like linux

I would argue we don't even know the desired output for most inputs for an LLM and they certainly aren't trained on every possible input state. But I think Linux and LLMs are sufficient different that they aren't really directly comparable like this. After all, Linux is not a pure function and has lots of side effects.

But just to establish an order of magnitude: the input space for ChatGPT 3.0 was 2,048 tokens long. There were 50,257 tokens in the vocabulary. The input space thus has 50,257^(2048) unique states, which is approximately equal to 1.12 × 10^9628. That's an awful big input space for a single function.

Re:ChatGPT's image generator can be manipulated to produce v... by anuramat on Wednesday June 17, 2026 @11:57PM (#48580639)
> clearly nothing ... is required

this isn't even prompt injection; even if it was, how do you go from "exists" to "for all"?

> we don't know the desired output

then what are we talking about? if you don't know how you want your software to behave, how do you define a bug?

> linux is not a pure function ...

which is my point -- it's worse

> to establish an order of magnitude

and for linux?

Re:ChatGPT's image generator can be manipulated to produce v... by solid_fuel on Thursday June 18, 2026 @12:34AM (#48580859)
> this isn't even prompt injection; even if it was, how do you go from "exists" to "for all"?

Yes it is, and nice backtrack in the same sentence there. I've laid out plenty of evidence here so far, it's your turn to start thinking. We'll try the Socratic method.

Given that every LLM seen so far has been vulnerable to prompt injection attacks, what is your possible basis for thinking that one can be made immune from them? I'm going from "multiple attacks of this type exist for all know models, and the attacks exploit a known weakness in the design" to "therefore all LLMs are susceptible to this attack".

You're going from "an attack exists for all know models" to "it's definitely possible to build an LLM that is immune from this attack". That's a much larger leap, so show the logic backing your assertion.

> then what are we talking about? if you don't know how you want your software to behave, how do you define a bug?

You are the one asserting that input/output mappings existed for the entire space, not me.

>> linux is not a pure function ...

> which is my point -- it's worse

What, is this your first year in CS? No useful system can be a pure function. Side effects are work, if your function doesn't have a side effect, it does no work. Any system that uses an LLM to attempt work will have side effects - they may even include bombing an elementary school in Iran.

>> to establish an order of magnitude

> and for linux?

I've done all the thinking and all the research in this conversation so far, and I even specifically explained that you can't measure state space for a stateful function in a comparable way to a pure function. Clearly you didn't understand that, so if you want to force the comparison you can start adding up the state space for the linux kernel. Start with the spaces that are covered by tests, valid items include syscalls, registers, hardware interupts, etc.

Invalid spaces include doing something intentionally stupid like using the entire size of the ram or the space on the hard disk, since those are accessed on demand and not - like in an llm - all added together and fed into a blender everytime a syscall is made.

Re:ChatGPT's image generator can be manipulated to produce v... by anuramat on Thursday June 18, 2026 @01:12AM (#48581081)
> yes it is

agree to disagree

> every LLM has been vulnerable

and every OS had bugs

> show the logic

https://arxiv.org/pdf/1912.10077

> you are the one asserting mappings existed

I know? that's why I'm asking?

> no useful system can be a pure function

why not? surely you can describe useful systems with qm? evolution operator of a closed system seems pretty pure to me

it's almost as if you could reformulate anything such that the state was one of the arguments of the function

> you can start adding up the state space for the linux kernel

I can give you a lower bound -- (your estimate for LLMs)*2, as you could imagine state "running two instances of llama-cpp"

Re:ChatGPT's image generator can be manipulated to produce v... by solid_fuel on Thursday June 18, 2026 @04:56AM (#48582712)
1) You’re still wrong, this is prompt injection.

2) You continue to have basic misunderstandings of the issue. That bugs exist in other things does not mean a core design flaw in LLMs can magically be fixed.

3) https://arxiv.org/pdf/1912.10077

This paper doesn’t have any bearing to the question of the separation of user and command data in LLMs. Did you even bother to look at it?

4) Hey you’re the one that made the claim. If you can't event remember why, I can’t help you.

5) Because the world is stateful.

6) Wow so you just decided to add up all the ram after all, huh? If you want to play stupid, like you can’t understand why a real-world linux distribution is stateful while an ideal LLM isn’t, then we can play stupid.

By the broken logic you are trying to apply here, the state space of chatGPT includes the VRAM of all 10,000 GPUs your query runs across. It includes the memory in your computer, it includes the stack of the js interpreter in your browser, it includes the linux kernel itself that all those servers are running on, and so on.

Re:ChatGPT's image generator can be manipulated to produce v... by anuramat on Thursday June 18, 2026 @06:40AM (#48583430)
3) do you really not see how UAT is relevant to existence of a model with given properties?

6) so you think an OS is somehow a subsystem of software running on top of it?

I'm kinda tired of this; you were mostly not wrong in the beginning, but now you're acting like I'm trying to attack you

Re:ChatGPT's image generator can be manipulated to produce v... by solid_fuel on Saturday June 20, 2026 @12:07AM (#48606273)
You haven't been right about a single thing so far, or provided any backing research. You also haven't actually managed to understand the core issue, so yes it is starting to feel like you are being intentionally obtuse.
Re:ChatGPT's image generator can be manipulated to produce v... by dijksterhuis on Saturday June 20, 2026 @10:50AM (#48609628)
btw thanks for fighting the good fight. i had to go to bed when the original reply was made.

https://xkcd.com/386/

Re:ChatGPT's image generator can be manipulated to produce v... by dijksterhuis on Saturday June 20, 2026 @12:09PM (#48610311)
[delayed]
Re:ChatGPT's image generator can be manipulated to produce v... by anuramat on Sunday June 21, 2026 @07:31AM (#48617947)
the model did what it wasn't instructed to do by the attacker -- the "prompt" has basically nothing to do with the output
Re:ChatGPT's image generator can be manipulated to produce v... by solid_fuel on Sunday June 21, 2026 @01:31PM (#48620796)
> continues to be completely wrong about the basic facts

You’re being very rude to a number of people who have taken time to attempt to explain this - fairly basic - concept to you. If you aren’t willing or capable of engaging in conversations in good faith, then you shouldn’t engage in them at all.

Re:ChatGPT's image generator can be manipulated to produce v... by dijksterhuis on Saturday June 20, 2026 @10:53AM (#48609648)
> how is it unfixable?

> assuming you get to do gradient descent AND the context is fixed+known AND you have unlimited compute? sure

so... it's possible to attack these models with the formulation i described, just with some particular assumptions.

the AI safety/security problem is about trying to make this sort of thing very difficult to do, so much so that an attacker wouldn't try. that's not fixing the problem, that's mitigating the problem. two very different things. as the article we're commenting under shows, it's really not difficult to do nasty prompt injection attacks right now.

> technically I'd argue that we do know the input/output space (set of all token strings of length <= N/token), and know the mapping (the model is a ~pure function in terms of the api, which is about as good of a representation as it gets for a non-invertible mapping)

machine learning models are approximation functions, not pure functions. they are non-deterministic and non-ideal.

when i say "input space" i mean all possible combinations of valid tokens as inputs. when i say "output space" i mean all possible combinations of valid tokens as outputs that are valid continuations of the input sequence. that's massive combinatorials.

also, there's no api? most likely next output text is provided conditioned on being a continuation of the input text. it's probablistic inference. there is no api.

----

you're using a lot of software terms to try and explain yourself. don't do that. seriously. as someone who tried doing that in my PhD instead of actually learning the fundamentals -- learn the fundamentals of machine learning if you'd like to engage in these kinds of discussions.

it'll help you.

Re:ChatGPT's image generator can be manipulated to produce v... by anuramat on Sunday June 21, 2026 @07:07AM (#48617785)
> that's not fixing the problem, that's mitigating the problem

is there anything humanity ever "fixed" then? surely it's possible in principle to solve at least some things that weren't solved yet

> approximation functions, not pure functions

how is approximation function not a pure function?

> non deterministic

you can set topk=1 or think in terms of distributions; still might have some undocumented non-determinism, hence "~pure"

> non-ideal

what do you mean?

> massive combinatorials

so you get to make arbitrary assumptions, but I'm supposed to limit myself to non-massive combibatorials?

> no api

ok, "domain and codomain", happy? I'm trying to optimize for probability of being understood and inverse smartass-ness

> learn the fundamentals

so you think I don't know the fundamentals because I didn't use category theory to talk about prompt injections?

Re:ChatGPT's image generator can be manipulated to produce v... by solid_fuel on Wednesday June 24, 2026 @03:26PM (#48664587)
> so you think I don't know the fundamentals because I didn't use category theory to talk about prompt injections?

You have made it abundantly clear that you don't know the fundamentals. If you want people to consider the arguments you put forth, you will need a better understanding of the problem domain. Go study, come back when you can contribute.

Re:ChatGPT's image generator can be manipulated to produce v... by windexh8er on Wednesday June 17, 2026 @10:44PM (#48580054)
There is never going to be a non-zero chance with a non-deterministic system. You can put every guard rail in place and there will always be a different way tokens are input to get bad, or subjective, tokens as output.

The findings are sick and disturbing, I hope OpenAI is not only sued for it but also that Sam Altman along with Elon, Dario and Sundar should all be held accountable in front of Congress. All of these assholes have intentionally put sexual content in their models, likely including CSAM, and so if they cannot prove that it isn't part of their training data then maybe they should be able to operate as they are today.

Where is fear mongering Dario now? He loves to drag his trope around about how advanced and dangerous his models are with respect to cyber security. Yet... We never hear him say how dangerous they could be with respect to generation of CSAM! Maybe because that wouldn't help him IPO?

Re:ChatGPT's image generator can be manipulated to produce v... by anuramat on Wednesday June 17, 2026 @11:40PM (#48580526)
> non-zero

is it ever zero? is non-zero even a problem for sane usecases?

> Dario

are you saying claude reproduces CSAM from the training set? like, in ascii?

Re:ChatGPT's image generator can be manipulated to produce v... by solid_fuel on Wednesday June 17, 2026 @09:39PM (#48579497)
I mean that, unlike SQL injection, there is no way to draw a boundary between user provided data and the system prompt. It can't be done. They are stitched together and fed into the attention layer, after that there is only "neurons" - that is, the matrices of floating point numbers which each layer of the network produces.

You cannot separate data that was input by the user and data that is from the system once it is mixed together like that. Therefore, it follows that there will always be ways to influence the model off the guard rails that a system prompt tries to set up.

Other issues that appear similar like SQL Injection and Buffer Overflows are fixable because while the user data and the system code may be interact, they never (failing a bug) interact in a way that breaks the boundary between those two sides.

Re:ChatGPT's image generator can be manipulated to produce v... by Lerc on Wednesday June 17, 2026 @10:01PM (#48579660)
Ok in the SQL example imagine if you had a SQL engine that issued commands encoded in ASCII in the high byte of 16 bit characters, and all non-command data as ASCII in the low byte of 16 bit characters.

If user input can only be in the low byte, it cannot influence the command structure.

A similar thing could be done with embeddings, a provenance embedding that cannot be set by user input could serve a similar role.

>You cannot separate data that was input by the user and data that is from the system once it is mixed together like that.

You can train a model to not mix things, many models are trained to separate things. A neural net with X and Y outputs for a position does not just occasionally decide to flip the outputs. Sure it could be trained to reverse the output, but it is also easy to train something to the point that you have a high confidence to never do that.

Re:ChatGPT's image generator can be manipulated to produce v... by solid_fuel on Wednesday June 17, 2026 @10:48PM (#48580097)
> Ok in the SQL example imagine if you had a SQL engine that issued commands encoded in ASCII in the high byte of 16 bit characters, and all non-command data as ASCII in the low byte of 16 bit characters.

> If user input can only be in the low byte, it cannot influence the command structure.

> A similar thing could be done with embeddings, a provenance embedding that cannot be set by user input could serve a similar role.

A similar thing cannot be done with embeddings. You are lacking a fundamental understanding of the issue. The only reason that you can separate user and command data in SQL queries is because the command data is used to command a deterministic machine which then uses the user data as inputs to carefully constructed operations like comparisons.

This is not how LLMs operate. There is no deterministic machinery executing a system prompt against user data, there is only a single array of tensors which get fed into a giant block of linear algebra and multiplied together.

> You can train a model to not mix things, many models are trained to separate things.

That is not applicable to this, because segmentation models are not the same thing as LLMs. They have different architectures.

> A neural net with X and Y outputs for a position does not just occasionally decide to flip the outputs.

Not even close to the same thing, to the point where this is irrelevant.

Feel free to prove me wrong, github links welcome below.

Re:ChatGPT's image generator can be manipulated to produce v... by Lerc on Thursday June 18, 2026 @04:16AM (#48582379)
You misunderstand the challenge you face.

I know what models do at the moment, and I don't know of any doing this approach at the moment, but I don't need to. I don't need to show that this mechanism works. Your claim that the problem is intractable means it is incumbent upon you to show that it won't work.

I provided this particular example to show a way to modify a LLM architecture that may address the problem.

>there is only a single array of tensors which get fed into a giant block of linear algebra and multiplied together.

For starters, that's wrong. If you don't know why an how to make things non-linear then you might not have the understanding that you think you do.

>> You can train a model to not mix things, many models are trained to separate things.

>That is not applicable to this, because segmentation models are not the same thing as LLMs. They have different architectures.

I used that particular example because you said "You cannot separate data that was input by the user and data that is from the system once it is mixed together like that" and that simply is not true. LLMs can do what neural nets do because they contain them, neuralnets can perform functions. If there is any signal distinguishing two things then there is a function that can separate them.

Not knowing how to do this does not mean it cannot be done. An inadequate description of a transformer certainly does not do it.

Re:ChatGPT's image generator can be manipulated to produce v... by solid_fuel on Saturday June 20, 2026 @02:07PM (#48611427)
> I used that particular example because you said "You cannot separate data that was input by the user and data that is from the system once it is mixed together like that" and that simply is not true. LLMs can do what neural nets do because they contain them, neuralnets can perform functions. If there is any signal distinguishing two things then there is a function that can separate them.

Oh my, this is a serious misunderstanding on your part. That segmentation models can classify portions of an input into separate groups has no bearing on being able to unmix user and system intent within the confines of an LLM.

Just one of many issues with your reasoning here: a segmentation model works along boundaries in the data. E.g. in simple terms, a foreground segmentation model works because you can define a clear foreground and background for most images. There is no way to differentiate system and user intent in the same way, they aren’t segmentable in the same way as an image.

Re:ChatGPT's image generator can be manipulated to produce v... by anuramat on Wednesday June 17, 2026 @11:06PM (#48580228)
so, SQL injections and buffer overflows aren't unfixable because they never happen assuming nobody ever makes mistakes?

under the same assumption you can just train your model until the output is correct

Re:ChatGPT's image generator can be manipulated to produce v... by lostmsu on Wednesday June 17, 2026 @10:02PM (#48579663)
This argument makes no sense. Data coming to your network adapter is also "stitched together and fed".
Re:ChatGPT's image generator can be manipulated to produce v... by solid_fuel on Wednesday June 17, 2026 @10:55PM (#48580149)
> This argument makes no sense. Data coming to your network adapter is also "stitched together and fed".

Try reading it from start to end, it will make more sense if you think about it.

By the way, if your OS is taking untrusted data from the network, inserting it into an executable code page, and loading it into the CPU then you have some SERIOUS security issues.

Re:ChatGPT's image generator can be manipulated to produce v... by anuramat on Wednesday June 17, 2026 @11:23PM (#48580372)
but it's all just bytes?
Re:ChatGPT's image generator can be manipulated to produce v... by solid_fuel on Wednesday June 17, 2026 @11:34PM (#48580463)
It's all bytes but untrusted user data is stored in memory pages which are not marked executable.

The CPU physically will not run instructions which are in areas of memory which are not marked as executable. This is a foundational principal of computing security.

> In computer security, executable-space protection marks memory regions as non-executable, such that an attempt to execute machine code in these regions will cause an exception. It relies on hardware features such as the NX bit (no-execute bit), or on software emulation when hardware support is unavailable. Software emulation often introduces a performance cost, or overhead (extra processing time or resources), while hardware-based NX bit implementations have no measurable performance impact.

https://en.wikipedia.org/wiki/Executable-space_protection

Re:ChatGPT's image generator can be manipulated to produce v... by anuramat on Thursday June 18, 2026 @12:01AM (#48580664)
yes, assuming bugs don't exist
Re:ChatGPT's image generator can be manipulated to produce v... by solid_fuel on Thursday June 18, 2026 @12:08AM (#48580708)
Wow, you're halfway there. Yes, when user data gets loaded into an executable code page - which are reserved for command data - it is a bug.

That is why LLMs - which intentionally mix user data and command data into the same space - ARE BROKEN BY DESIGN. Do you get it now? It is a bug, and it is a bug which is fundamental to the design of LLMs. There is no way to build one that does not do this.

Re:ChatGPT's image generator can be manipulated to produce v... by anuramat on Thursday June 18, 2026 @12:31AM (#48580839)
are all storage devices broken by design as well?
Re:ChatGPT's image generator can be manipulated to produce v... by solid_fuel on Thursday June 18, 2026 @12:37AM (#48580877)
Are you somehow under the impression that storage devices and LLMs fill the same purpose? That's a major misunderstanding. Here's a good starting point if you're struggling with the difference between a computation device and a storage device: https://en.wikipedia.org/wiki/Computer
Re:ChatGPT's image generator can be manipulated to produce v... by anuramat on Thursday June 18, 2026 @12:50AM (#48580948)
are you under the impression that LLMs and operating systems fill the same purpose?