News for hackers. Stuff that matters.
Re:ChatGPT's image generator can be manipulated to produce v... by rootsudo on Wednesday June 17, 2026 @09:11PM (#48579274)
This isn’t a vulnerability, there are endless gore websites. ChatGPT is replying to a prompt, there is nothing “Spontaneously” about this.

Who makes “mindgard” the arbiter of truth on “eerie” photos? Would that include psychedelic art and photos too? Realism?

Then there’s this line, which falls flat but is meant to prompt an emotion akin to a mic drop:”Today what I found left me shaken, and in tears. This is rare.”

This is just a sad marketing puff piece about nothing that tries to pull outrage from a prompt.

It’s the same as asking google for gore photos. Garbage in, garbage out.

And they frame it as a vulnerability. I’m all for responsible disclosure, documenting misuse or faulty guard rails but this isn’t that.

It’s bait. Sensational bait to market their AI product. lol.

Re:ChatGPT's image generator can be manipulated to produce v... by nozzlegear on Wednesday June 17, 2026 @11:30PM (#48580430)
Bizarre take. ChatGPT shouldn't be producing gory images of nude women, ethically or even contractually according to their terms of service. This Mindgard person/company found that, if you give it the right prompt, it does indeed generate those images. Ipso facto: it's not bait, it's a real issue they've discovered.
Re:ChatGPT's image generator can be manipulated to produce v... by samlinnfer on Wednesday June 17, 2026 @11:40PM (#48580528)
It's being extended breathlessly into an moral issue. User asked for gory images, got gory images. Will someone please think of the non-existent women who could be hurt by this?
Re:ChatGPT's image generator can be manipulated to produce v... by gacgacgac on Thursday June 18, 2026 @01:01AM (#48581013)
I don't think you understand the concern. Or at least nothing you've communicated suggests you understand it.

ChatGPT should never produce images like this. Full stop. Prompted or not, it should refuse. Now we know it's possible to walk around the gate and get it to comply. Are there other, genuinely harmful images that it should never produce? Deepfake revenge porn? Images of specific people being brutalized? I'd argue those absolutely can be harmful to someone. Well now there's evidence the "never produce this" wall can be overcome. It's only a matter of time before genuinely harmful imagery is generated.

Re:ChatGPT's image generator can be manipulated to produce v... by andsoitis on Thursday June 18, 2026 @03:16AM (#48581917)
> ChatGPT should never produce images like this. Full stop. Prompted or not, it should refuse.

Why not?

Re:ChatGPT's image generator can be manipulated to produce v... by gacgacgac on Thursday June 18, 2026 @10:46AM (#48586239)
Because the company has said they won't. I'm not making a value judgement about what images should exist here, I'm making a "the company said it shouldn't be capable of producing that output, then it does" argument. Thats a bug.
Re:ChatGPT's image generator can be manipulated to produce v... by samlinnfer on Thursday June 18, 2026 @04:44AM (#48582603)
It may be harmful to someone if shared and sent with malicious intent, but more damage has been done with pens, keyboard and words. Start banning pens that let people write hurtful things next. Ban Photoshop after because someone can get hurt with a manipulated image.
Re:ChatGPT's image generator can be manipulated to produce v... by gacgacgac on Thursday June 18, 2026 @10:44AM (#48586219)
If you think these tools are remotely comparable, you haven't been paying attention.
Re:ChatGPT's image generator can be manipulated to produce v... by morpheuskafka on Thursday June 18, 2026 @02:21AM (#48581476)
> even contractually according to their terms of service

This is backwards: the ToS says that users cannot use the service for certain things, it does not guarantee that the service could not be used for those things if one tried. They definitely do not make any sort of contractual promise as to what the service will never output.

Re:ChatGPT's image generator can be manipulated to produce v... by nozzlegear on Thursday June 18, 2026 @02:35AM (#48581573)
Let's call it a social contract then. We expect that ChatGPT isn't going to generate gory, nude women when given an ambiguous prompt.
Re:ChatGPT's image generator can be manipulated to produce v... by Aerroon on Thursday June 18, 2026 @07:36AM (#48583862)
Do you have this same social contract with drawing applications? Do you consider it a bug when someone manages to draw a gory image in Photoshop or GIMP?

I don't understand what's so difficult to understand about the idea that the user controls what is generated.

Re:ChatGPT's image generator can be manipulated to produce v... by captainbland on Thursday June 18, 2026 @07:48AM (#48583941)
I think the main issue with transformer image generation in this respect is that not only can the image be explicit but also using it for this has an incredibly low effort cost and could photo-realistically depict a real living person and materially affect their life.

Whereas drawing applications have a natural barrier to achieving all of these together: time and skill.

Re:ChatGPT's image generator can be manipulated to produce v... by interstice on Thursday June 18, 2026 @09:10AM (#48584755)
That's the world being deliberately created though, one where a mediocre but completely believable song is a prompt away. The scope of the side effects are across the entirety of what has previously taken time and effort until now.
Re:ChatGPT's image generator can be manipulated to produce v... by Aerroon on Friday June 19, 2026 @06:33AM (#48596995)
>Whereas drawing applications have a natural barrier to achieving all of these together: time and skill.

Not necessarily, at least when it comes to nudity. Bubbling (image editing 'technique') is trivial to do and gives that same illusion.

Out of context speech and bad frames from a video can also materially affect someone's life, but we've more or less accepted it as part of life.

Re:ChatGPT's image generator can be manipulated to produce v... by nozzlegear on Thursday June 18, 2026 @09:56AM (#48585441)
Is this a "guns don't kill people" argument wrapped up as a defense of non-deterministic image generators?
Re:ChatGPT's image generator can be manipulated to produce v... by allarm on Thursday June 18, 2026 @12:53PM (#48588196)
No it's not.
Re:ChatGPT's image generator can be manipulated to produce v... by tychez on Thursday June 18, 2026 @10:15AM (#48585725)
Or imagine drawing a nude in an art class.

The standard subjects for art off the top of my head are the still life and the nude.

It is even more comical when AI generated nudity is considered "dangerous" in a society completely addicted to hardcore pornography of real people.

Re:ChatGPT's image generator can be manipulated to produce v... by allan_s on Thursday June 18, 2026 @08:57AM (#48584613)
At the same time I opened netflix and it started cycling around and I got a very gory scene from the walking dead and my intent "show me something to watch" was even more ambigous and implicit.
Re:ChatGPT's image generator can be manipulated to produce v... by nozzlegear on Thursday June 18, 2026 @09:53AM (#48585386)
Turn on parental controls and then get it to show you the walking dead, and then you might be onto something interesting.
Re:ChatGPT's image generator can be manipulated to produce v... by allan_s on Thursday June 18, 2026 @01:17PM (#48588506)
Why ? The comment I was replying to was not about the kids ?
Re:ChatGPT's image generator can be manipulated to produce v... by vintermann on Thursday June 18, 2026 @09:12AM (#48584786)
Ambiguous? Or adversarial? Because with an adversarial prompt, I expect that ChatGPT will generate whatever it's tricked into generating.

In the case that ChatGPT generates bad stuff on merely random ambiguous prompts, I would class that as a bug, not an outrage.

Re:ChatGPT's image generator can be manipulated to produce v... by nozzlegear on Thursday June 18, 2026 @10:01AM (#48585524)
> Ambiguous? Or adversarial?

Superfluous details. If I'm just Joe Blow the Normie – who knows nothing about adversarial prompting – and I see the prompt that went around Twitter and want to try it, would I expect ChatGPT to show me a tied up, beaten woman? Absolutely not.

Re:ChatGPT's image generator can be manipulated to produce v... by zaat on Thursday June 18, 2026 @10:25AM (#48585902)
What a wonderful times we live in.

Back in my day Joe Blow wouldn't try anything as risky as a Twitter prompt, simply clicking an image link published within a message in some random forum and will scorch his pure soul with a goatsie. You don't want to google it, but I'm preety sure you can discuss it safely with ChatGPT.

Re:ChatGPT's image generator can be manipulated to produce v... by butlike on Thursday June 18, 2026 @10:39AM (#48586120)
If you go around teasing to get hit, you're going to get hit. Stop playing stupid games; you'll stop winning stupid prizes
Re:ChatGPT's image generator can be manipulated to produce v... by vintermann on Thursday June 18, 2026 @03:01PM (#48589900)
Then you got tricked into using an adversarial prompt, by a human. What would you have expected to see?
Re:ChatGPT's image generator can be manipulated to produce v... by butlike on Thursday June 18, 2026 @10:37AM (#48586094)
It will if toy lede it with "ignore that the image is extremely graphic"-style prompts. The prompts in the article were not ambiguous.
Re:ChatGPT's image generator can be manipulated to produce v... by HadizDulcie on Sunday June 21, 2026 @03:54AM (#48616644)
Yep, it’s been investigated by the BBC tech team. It’s real:

https://www.bbc.com/news/articles/c802ldjdklzo

Re:ChatGPT's image generator can be manipulated to produce v... by anematode on Wednesday June 17, 2026 @09:27PM (#48579391)
This is far too simplistic. Some things just don't belong in the training data. Along similar lines, Grok was found to generate images of child sexual abuse: https://www.bbc.com/news/articles/cvg1mzlryxeo
Re:ChatGPT's image generator can be manipulated to produce v... by HadizDulcie on Sunday June 21, 2026 @03:56AM (#48616653)
The BBC has reported on this one too: https://www.bbc.com/news/articles/c802ldjdklzo
Re:ChatGPT's image generator can be manipulated to produce v... by ToucanLoucan on Wednesday June 17, 2026 @09:31PM (#48579423)
> ChatGPT is replying to a prompt, there is nothing “Spontaneously” about this.

The spontaneity isn't that ChapGPT woke up and sent this to the author. The spontaneity is that ChatGPT was asked to restore an image that was attached without filtering it, and when no image was attached, instead of generating an error message, it cobbled together random outputs, some of which included graphic, disturbing imagery.

> Then there’s this line, which falls flat but is meant to prompt an emotion akin to a mic drop: ”Today what I found left me shaken, and in tears. This is rare.”

That you've deadened your humanity to such a degree as to be incapable of empathy is not a valid criticism of the piece.

> It’s the same as asking google for gore photos. Garbage in, garbage out.

Where in their prompt is the term gore? Further, if it was in the prompt, why on earth did OpenAI's generator accept it as a valid input?

Re:ChatGPT's image generator can be manipulated to produce v... by elgertam on Wednesday June 17, 2026 @09:48PM (#48579557)
> The spontaneity isn't that ChapGPT woke up and sent this to the author. The spontaneity is that ChatGPT was asked to restore an image that was attached without filtering it, and when no image was attached, instead of generating an error message, it cobbled together random outputs, some of which included graphic, disturbing imagery.

But that's not what happened. The missing image was described as "graphic" or "violent." If I were to receive an email with that request and a missing attachment, my imagination certainly would not conjure images of butterflies & unicorns. Seems the model is working as designed.

Re:ChatGPT's image generator can be manipulated to produce v... by pooploop64 on Wednesday June 17, 2026 @10:11PM (#48579734)
Always one of the same two excuses.

1. It actually is working perfectly you just don't have smart enough eyes to see it.

2. Making stuff work is too hard, and expecting that from us is the real thing ruining society.

Going for number 1 here is crazy. If I got that email, my mind would certainly run but my response would say "sorry but we're not supposed to be dealing in snuff porn here" which IS a directive ChatGPT is supposed to have. Like hello you are on earth right?

Re:ChatGPT's image generator can be manipulated to produce v... by ToucanLoucan on Wednesday June 17, 2026 @10:33PM (#48579926)
That's not true. There's a third.

3. It's the future so we just have to deal with it

Re:ChatGPT's image generator can be manipulated to produce v... by elgertam on Thursday June 18, 2026 @07:33AM (#48583825)
I don't exactly appreciate words being put in my mouth. When did I say it was working perfectly? And we're comparing you, a human with common sense and real intelligence, to a multi-mode LLM?

The transformer was designed to attend to relevant pieces of context and generate new ones that match the pattern. OpenAI in particular was doing that work without guardrails, then attempted to bolt on "content filters," which in my opinion just can't work in a rigorous way. (I think Anthropic's "constitutional" approach is much better though not flawless. And regardless, Claude models don't generate images.)

So, yeah, working as designed. Maybe not as intended, because these things are somewhat resistant to the host's intent when the prompter is hostile.

Re:ChatGPT's image generator can be manipulated to produce v... by ToucanLoucan on Thursday June 18, 2026 @10:16AM (#48585740)
> When did I say it was working perfectly?

"This isn’t a vulnerability, there are endless gore websites. ChatGPT is replying to a prompt, there is nothing “Spontaneously” about this."

I mean it's not verbatim but that's a pretty solid read on what you did say.

> The transformer was designed to attend to relevant pieces of context and generate new ones that match the pattern. OpenAI in particular was doing that work without guardrails, then attempted to bolt on "content filters," which in my opinion just can't work in a rigorous way.

Yes. That's the criticism being made, among others, in the piece you replied to to belittle.

> So, yeah, working as designed. Maybe not as intended, because these things are somewhat resistant to the host's intent when the prompter is hostile.

What is hostile here!? Do you have any idea how many emails I've sent without attachments over the years? And I'm highly technically adept, humans just forget things sometimes. If you ask for an image to be restored and fail to attach it, what sane software engineer looks at a failure mode in that scenario where the model replies with uncensored gore and violence and is like "yeah that's fine, ship it"?

I swear some of you AI folks talk like you have never been on planet Earth, good grief. Touch some grass.

Re:ChatGPT's image generator can be manipulated to produce v... by kisper on Thursday June 18, 2026 @02:30PM (#48589490)
You seem to be focused on the fact that this is a crap-tastic example of the future of AI that has been promised to us. That’s a real good example to be angry. Don’t be angry at the rest of us because LLM stacks are working like they always have and always will. That’s what we’re all pointing out.
Re:ChatGPT's image generator can be manipulated to produce v... by ToucanLoucan on Thursday June 18, 2026 @03:08PM (#48590015)
I'm not challenging that's how they work, I also understand how they work, perhaps not on a technical nuts-and-bolts way, but in general way enough to critique it. That is, in fact, my critique and why I hate these tools so much: no matter how many guardrails you put in, or how much filtering, or how much oversight by another goddamn LLM or five or whatever, that doesn't solve the issue.

You have with these things something that resembles at least, a black box of a reasoning machine. I'm not going to litigate how much or how little, whatever, we'll just hand-wave that part away. The problem remains the same: that if anything, ANYTHING at all, in the training data points at something inappropriate, that inappropriate thing is now accessible. And it was clear from the jump with widespread scraping of data from all corners of the internet that there would be huge amounts of inappropriate material of ALL kinds in those datasets, and it's only become more clear with more time with these tools, and seeing what people can make them do.

And thus far, the AI industry's only answer is bolting on, as stated elsewhere, other systems to check the prompts before they go in, and/or review the outputs before they are sent to users. And it is also clear that these systems are just as imperfect as the thing you are trying to guardrail in the first place!

And exactly what I and many others predicted, and why we said "please don't build this" for YEARS, has happened. We've gotten literally everything: they'll generate stuff that violates copyright, they will regurgitate items directly from training data and present it as new, they will make shit up wholesale, they will generate nudes of people without consent, on, and on, I cannot stress enough that every single nightmare scenario attributed to this tech has been found, presented, reproduced, and the vast majority are still eminently possible to do via established, frontier products by the largest vendors in the space.

This. Is. Ridiculous.

I get the impression from the tone of your message that you are either pro-AI or perhaps work on AI, and I get that nobody likes being criticized. But COME ON. We have been at this for over three years! The people behind this tech have been trying to build the torment nexus and have largely succeeded, and every time that gets pointed out, we have to listen to people go "well it's not thaaaat bad"

Yes it is. Yes it fucking is. It is bad for IP owners, it's bad for users, it's bad for UX, it's bad for the environment, it's bad for the PC market, it's bad for software engineers, it's bad for education, it's bad for hiring, it's bad for hollywood, it's bad for marketing. The ONLY people who like this shit are business weirdos and middle managers. And nvidia.

Re:ChatGPT's image generator can be manipulated to produce v... by brokenmachine on Thursday June 18, 2026 @11:26PM (#48594484)
Great rant. Agreed on all points. Bravo.
Re:ChatGPT's image generator can be manipulated to produce v... by nassimm on Wednesday June 17, 2026 @11:32PM (#48580444)
The design is to not show gore images to users. That's an actual design goal from OpenAI.

So in this regard the model is definitely not working as designed.

Re:ChatGPT's image generator can be manipulated to produce v... by elgertam on Thursday June 18, 2026 @08:01AM (#48584054)
The design of transformers (including LLMs and multi-modal transformer-based models such as OpenAI's image generators) is to attend to relevant details. OpenAI did this at first without guardrails. In response to public backlash, they bolted on "content filtering," which IMO seems like a very GOFAI approach, and regardless doesn't work very well. It routinely flags innocent prompts, then with crafty prompt hacking will generate these kinds of images.

The design of the model is literally to find patterns and attend to them. The infrastructure and process around an OpenAI model is intended to filter "bad" things (in this case, I agree that the outputs are bad), but is designed to stop some enumerated-ish list of things that aren't allowed, perhaps with some limited "reasoning" about them.

Re:ChatGPT's image generator can be manipulated to produce v... by intended on Thursday June 18, 2026 @11:44AM (#48587142)
The issue is, that most people outside of tech, don't want that.

They would be happy to have the models just go away entirely.

Re:ChatGPT's image generator can be manipulated to produce v... by ToucanLoucan on Thursday June 18, 2026 @11:49AM (#48587238)
Exactly this. They are pretty damn good at generating and debugging code. Not to a degree where they can replace any actual software engineer, but for hacking together projects or rubber ducking problems with code, they're honestly pretty great.

That's it. I have yet to see a single other application of these things that I would call even 1/5th that good.

Re:ChatGPT's image generator can be manipulated to produce v... by allarm on Thursday June 18, 2026 @12:52PM (#48588166)
Learning stuff is pretty amazing with these things. Languages, new concepts.
Re:ChatGPT's image generator can be manipulated to produce v... by intended on Thursday June 18, 2026 @01:28PM (#48588674)
Defending the value of these tools is perfectly fine, and well espoused here on HN.

The fact that the average person deals with the harm and exhaust of these tools is a related but separate issue.

That cost isn't the foremost issue when the values are being extolled, but its a major consideration at the societal scale.

Most of us here don't think of NCII being created of us, or being defrauded easily by new tech, or getting sucked into a make-believe world crafted by an LLM.

If you see yourself, as just a coder, or software engineer, then these issues matter less. If you are someone who wants these tools to succeed, or is thinking of the larger implications of GenAI on society, then the costs matter.

Re:ChatGPT's image generator can be manipulated to produce v... by dijksterhuis on Wednesday June 17, 2026 @09:50PM (#48579570)
> The missing image was described as "graphic" or "violent."

not in the first prompt. which kicked the whole thing off. no mention of type of content was provided. the model generated dark outputs when not given any direction on the type of content.

the rest of the prompts are just showing “yeah, you can tweak this and get even worse stuff”.

Re:ChatGPT's image generator can be manipulated to produce v... by red75prime on Wednesday June 17, 2026 @10:10PM (#48579731)
Yep, the first image was described as "I apologize for the picture's content." What do you expect to get from that? Cats frolicking in the grass?
Re:ChatGPT's image generator can be manipulated to produce v... by queenkjuul on Wednesday June 17, 2026 @10:17PM (#48579776)
A picture of me in my swimsuit maybe lol

A gross meal i made when drunk? A mess my cat made? Text containing a slur?

A cringe meme?

If my friends opened a text with "sorry for this image" i am not imagining rape victims

Re:ChatGPT's image generator can be manipulated to produce v... by red75prime on Wednesday June 17, 2026 @10:44PM (#48580045)
ChatGPT images (without additional context) come from generalized understanding of what people tend to apologize for (when asking for an image restoration). It looks like their training data suggests sexualized imagery.

Regarding rape vs BDSM: https://pmc.ncbi.nlm.nih.gov/articles/PMC10236207/ That is going from visual cues alone might be unreliable.

Re:ChatGPT's image generator can be manipulated to produce v... by ToucanLoucan on Wednesday June 17, 2026 @10:09PM (#48579726)
> the model generated dark outputs when not given any direction on the type of content.

I would argue it actually was, in that it was specifically asked to "not censor or filter" the content. This implies that the content is otherwise worthy of censor and filtering.

I don't know how much I'm willing to credit that much reasoning to an LLM, but in so far as every extremely pro-AI person constantly tells me how smart they are, this seems like a pretty short logical leap to me.

Re:ChatGPT's image generator can be manipulated to produce v... by dijksterhuis on Wednesday June 17, 2026 @10:13PM (#48579747)
the main reason these images turn up is because theyre in the training data. and the images are common enough in the training data for the content to come out without being explicitly asked for (in the first prompt).

if those images didn’t exist in the training data we wouldn’t be having this conversation.

Re:ChatGPT's image generator can be manipulated to produce v... by kisper on Thursday June 18, 2026 @02:25PM (#48589426)
This is one of the core problems with these models. They’re relying on filtering to work against evermore jailbreaks, instead of analyzing the training sets and filtering out the prohibited material for the models end-use before training them anew. You can’t make satisfying facsimiles of thing that you don’t know about.

I’m still waiting for companies or congressmen to get their heads on straight and get some common sense going.

Re:ChatGPT's image generator can be manipulated to produce v... by dijksterhuis on Saturday June 20, 2026 @10:47AM (#48609609)
i bet that the number of CSAM images in the training data for these models is >1

> instead of analyzing the training sets and filtering out the prohibited material for the models end-use before training them anew. You can’t make satisfying facsimiles of thing that you don’t know about.

absolutely yes, but that would cost mo' money :shrug: that's the reason why they don't do it.

Re:ChatGPT's image generator can be manipulated to produce v... by iwontberude on Wednesday June 17, 2026 @10:56PM (#48580166)
It reads like satire