I want image generators to generate what I ask them and not alter my query into something else.

It's deeply shameful that billions of dollars and the hard work of incredibly smart people is mangled for a 'feature' that most end users don't even want and can't turn off.

This is not a one off, it keeps happening with generative AI all the time. Silent prompt injections are visible for now with jailbreaks but who knows what level of stupidity goes on during training?

Look at this example from the Würstchen paper (which stable cascade is based on):

>This work uses the LAION 5-B dataset...

>As an additional precaution, we aggressively filter the dataset to 1.76% of its original size, to reduce the risk of harmful content being accidentally present (see Appendix G).

> Silent prompt injections

That’s the crux of what’s so off-putting about this whole thing. If Google or OpenAI told you your query was to be prepended with XYZ instructions, you could calibrate your expectations correctly. But they don’t want you to know they’re doing that.

This sounds bit entitled. It is just service of private company.

If it's not going to give you what it's promising, which is generating images based on the prompts you provide it, it's a poor service. I think it might make more sense to try determine whether it's appropriate or not to inject ethnic or gender diversity into the prompt, rather than doing so without regard for context. I'm not categorically opposed to compensating for biases in the training data, but this was done very clumsily at best.

Yes, and I want the services I buy from private companies to do certain things.

Is it equally entitled to ask for a search engine which brings answers related to my query?

Not to be overly cynical, but this seems like it's the likely outcome in the medium-term.

Billions of dollars worth of data and manhours could only be justified for something that could turn a profit, and the obvious way an advertising company like Google could make money off a prompt handler like this would be "sponsored" prompts. (i.e. if I ask for images of Ben Franklin and Coke was bidding, then here's Ben Franklin drinking a refreshing diet coke)

As far as we know, there are no photos of Vikings. It's reasonable for someone to use AI for learning about their appearance. If working as intended, it should be as reliable as reading a long description of Vikings on Wikipedia.

We have tons of viking material culture you can access directly without the AI layer.

AI as learning tool here feels misplaced to me.

what's the point of image generators then? what if i want to put vikings in a certain setting, in a certain artistic style?

Than put that into the prompt explicitly instead of relying on Google, OpenAI, or whatever to add "racially ambiguous"

Then specify that in your prompt. "... in the style of ..." or "... in a ... setting".

The point is that those modifications should be reliable, so if you want a viking man/woman or an asian/african/greek viking then adding those modifiers should all just work.

The problem is more that it refuses to make images of white people than the accuracy of the historical ones.

I don't know what you mean by "represent actual history". I don't think anyone believes that AI output is supposed to replace first-party historical sources.

But we are trying to create a tool where we can ask it questions and it gives us answers. It would be nice if it tried to make the answers accurate.

To which they reply "well you weren't actually there and this is art so there are no rules." It's all so tiresome.

In your favour is the fact that AI can "hallucinate", and generate realistic, but false information. So that does raise the question "why are you using AI when seeking factual reference material?".

However on the other hand that is a misuse of AI, since we already know that hallucinations exist, are common, and that AI output must be verified by a human.

So as a counterpoint, there are sound reasons for using AI to generate images based on history. The same reasons are why we use illustrations to demonstrate ideas where there is no photographic record.

A straightforward example is visualising the lifetime/lifestyle of long past historical figures.

ideological testing, we got to know how they cooked the model

It's as if Google believes their higher principle is something other than serving customers and making money. They haven't been able to push out a new successful product in 10+ years. This doesn't bode well for them in the future.

I blame that decade of near zero interest rates. Companies could post record profits without working for them. I think in the coming years we will discover that that event functionally broke many companies.

Indeed. Check this: https://mindmatters.ai/2022/01/the-strange-story-of-googles-...

You're right we should ban images of history altogether. Infact I think we should ban written accounts too. We should go back to the oral historic tradition of the ancient Greeks

He did not say he wanted to ban images, that is an exaggeration. I see the danger as polluting the historical record with fake images (even as memes/jokes), and spreading wrong preconceptions now backed by real-looking images. This is all under the assumptions there are no bad actors, which makes it even worse. I would say; don't ban it, but you morally just shouldn't do it.

The real danger is that this anti-racism starts a justified round of new racism.

By lowering standards for black doctors do you think anyone in their right mind would pick black doctors? No I want the fat old jew. I know no one put him in the hospital to fill out a quota.

Exactly, and as we all know all ancient Greeks were people of color, just like Cleopatra.

Woah, no one said that but you.

> why are we using image generators to represent actual history?

That’s what a movie going to be in the future. People are going to prompt characters that AI will animate.

Ah, good point. I'll just use the actual photograph of George Washington boxing a kangaroo.

Ah. So we can trust AI to answer truthfully about history (and other issues), but we can't expect it to generate images for that same history, got it.

Any other specific things we should not expect from AI or shouldn't ask AI to do?

No, I don't think you can trust AI to answer correctly, ever. I've seen it confidently hallucinate, so I would always check what it says against other, more static, sources. The same if I'm reading from an author who includes a lot of mistakes in his books: I might still find them interesting and usefull, but I will want to double-check the key facts before I quote them to others.

Saying this is no different than saying you can't trust computers, ever, because they were (very) unreliable in the 50s and early 60s. We've been doing "good" generative AI for around 5 years, there is still much to improve until it reaches the reliability of other information sources like Wikipedia and Britannica.

> Saying this is no different than saying you can't trust computers, ever, because they were (very) unreliable in the 50s and early 60s

This seems completely reasonable to me. I still don't trust computers.

No, you should not trust AI to answer truthfully about anything. It often will, but it is well known that LLMs hallucinate. Verify all facts. In all things, really, but especially from AI.

I think we're not even close technologically, but creating historically accurate (based on the current level of knowledge humanity has of history) depictions, environments and so on is, to me, one of the most _fascinating_ applications.

Insane amounts of research go into creating historical movies, games etc that are serious about getting it right. But to try and please everyone, they take lots of liberties, because they're creating a product for the masses. For that very same reason, we get tons of historical depictions of New York and London, but none of the medium sized city where I live.

The effort/cost that goes into historical accuracy is not reasonable without catering to the mass market, so it seems like a conundrum only lots of free time for a lot of people or automation could possibly break.

Not holding my breath that it's ever going to be technically possible, but boy do I see the appeal!

It should generate the image I ask for. As seen, if it explicitly refuses to generate images of white people and blathers on about problematic this-and-that as its "justification", there is a deep issue at hand.