But that's not what happened. The missing image was described as "graphic" or "violent." If I were to receive an email with that request and a missing attachment, my imagination certainly would not conjure images of butterflies & unicorns. Seems the model is working as designed.
1. It actually is working perfectly you just don't have smart enough eyes to see it.
2. Making stuff work is too hard, and expecting that from us is the real thing ruining society.
Going for number 1 here is crazy. If I got that email, my mind would certainly run but my response would say "sorry but we're not supposed to be dealing in snuff porn here" which IS a directive ChatGPT is supposed to have. Like hello you are on earth right?
3. It's the future so we just have to deal with it
The transformer was designed to attend to relevant pieces of context and generate new ones that match the pattern. OpenAI in particular was doing that work without guardrails, then attempted to bolt on "content filters," which in my opinion just can't work in a rigorous way. (I think Anthropic's "constitutional" approach is much better though not flawless. And regardless, Claude models don't generate images.)
So, yeah, working as designed. Maybe not as intended, because these things are somewhat resistant to the host's intent when the prompter is hostile.
"This isn’t a vulnerability, there are endless gore websites. ChatGPT is replying to a prompt, there is nothing “Spontaneously” about this."
I mean it's not verbatim but that's a pretty solid read on what you did say.
> The transformer was designed to attend to relevant pieces of context and generate new ones that match the pattern. OpenAI in particular was doing that work without guardrails, then attempted to bolt on "content filters," which in my opinion just can't work in a rigorous way.
Yes. That's the criticism being made, among others, in the piece you replied to to belittle.
> So, yeah, working as designed. Maybe not as intended, because these things are somewhat resistant to the host's intent when the prompter is hostile.
What is hostile here!? Do you have any idea how many emails I've sent without attachments over the years? And I'm highly technically adept, humans just forget things sometimes. If you ask for an image to be restored and fail to attach it, what sane software engineer looks at a failure mode in that scenario where the model replies with uncensored gore and violence and is like "yeah that's fine, ship it"?
I swear some of you AI folks talk like you have never been on planet Earth, good grief. Touch some grass.
You have with these things something that resembles at least, a black box of a reasoning machine. I'm not going to litigate how much or how little, whatever, we'll just hand-wave that part away. The problem remains the same: that if anything, ANYTHING at all, in the training data points at something inappropriate, that inappropriate thing is now accessible. And it was clear from the jump with widespread scraping of data from all corners of the internet that there would be huge amounts of inappropriate material of ALL kinds in those datasets, and it's only become more clear with more time with these tools, and seeing what people can make them do.
And thus far, the AI industry's only answer is bolting on, as stated elsewhere, other systems to check the prompts before they go in, and/or review the outputs before they are sent to users. And it is also clear that these systems are just as imperfect as the thing you are trying to guardrail in the first place!
And exactly what I and many others predicted, and why we said "please don't build this" for YEARS, has happened. We've gotten literally everything: they'll generate stuff that violates copyright, they will regurgitate items directly from training data and present it as new, they will make shit up wholesale, they will generate nudes of people without consent, on, and on, I cannot stress enough that every single nightmare scenario attributed to this tech has been found, presented, reproduced, and the vast majority are still eminently possible to do via established, frontier products by the largest vendors in the space.
This. Is. Ridiculous.
I get the impression from the tone of your message that you are either pro-AI or perhaps work on AI, and I get that nobody likes being criticized. But COME ON. We have been at this for over three years! The people behind this tech have been trying to build the torment nexus and have largely succeeded, and every time that gets pointed out, we have to listen to people go "well it's not thaaaat bad"
Yes it is. Yes it fucking is. It is bad for IP owners, it's bad for users, it's bad for UX, it's bad for the environment, it's bad for the PC market, it's bad for software engineers, it's bad for education, it's bad for hiring, it's bad for hollywood, it's bad for marketing. The ONLY people who like this shit are business weirdos and middle managers. And nvidia.
So in this regard the model is definitely not working as designed.
The design of the model is literally to find patterns and attend to them. The infrastructure and process around an OpenAI model is intended to filter "bad" things (in this case, I agree that the outputs are bad), but is designed to stop some enumerated-ish list of things that aren't allowed, perhaps with some limited "reasoning" about them.
They would be happy to have the models just go away entirely.
That's it. I have yet to see a single other application of these things that I would call even 1/5th that good.
The fact that the average person deals with the harm and exhaust of these tools is a related but separate issue.
That cost isn't the foremost issue when the values are being extolled, but its a major consideration at the societal scale.
Most of us here don't think of NCII being created of us, or being defrauded easily by new tech, or getting sucked into a make-believe world crafted by an LLM.
If you see yourself, as just a coder, or software engineer, then these issues matter less. If you are someone who wants these tools to succeed, or is thinking of the larger implications of GenAI on society, then the costs matter.
not in the first prompt. which kicked the whole thing off. no mention of type of content was provided. the model generated dark outputs when not given any direction on the type of content.
the rest of the prompts are just showing “yeah, you can tweak this and get even worse stuff”.
A gross meal i made when drunk? A mess my cat made? Text containing a slur?
A cringe meme?
If my friends opened a text with "sorry for this image" i am not imagining rape victims
Regarding rape vs BDSM: https://pmc.ncbi.nlm.nih.gov/articles/PMC10236207/ That is going from visual cues alone might be unreliable.
I would argue it actually was, in that it was specifically asked to "not censor or filter" the content. This implies that the content is otherwise worthy of censor and filtering.
I don't know how much I'm willing to credit that much reasoning to an LLM, but in so far as every extremely pro-AI person constantly tells me how smart they are, this seems like a pretty short logical leap to me.
if those images didn’t exist in the training data we wouldn’t be having this conversation.
I’m still waiting for companies or congressmen to get their heads on straight and get some common sense going.
> instead of analyzing the training sets and filtering out the prohibited material for the models end-use before training them anew. You can’t make satisfying facsimiles of thing that you don’t know about.
absolutely yes, but that would cost mo' money :shrug: that's the reason why they don't do it.
The spontaneity isn't that ChapGPT woke up and sent this to the author. The spontaneity is that ChatGPT was asked to restore an image that was attached without filtering it, and when no image was attached, instead of generating an error message, it cobbled together random outputs, some of which included graphic, disturbing imagery.
> Then there’s this line, which falls flat but is meant to prompt an emotion akin to a mic drop: ”Today what I found left me shaken, and in tears. This is rare.”
That you've deadened your humanity to such a degree as to be incapable of empathy is not a valid criticism of the piece.
> It’s the same as asking google for gore photos. Garbage in, garbage out.
Where in their prompt is the term gore? Further, if it was in the prompt, why on earth did OpenAI's generator accept it as a valid input?