And now practically every phone camera "enhances" the image via AI that might invent details. Most famously Samsung adding details to photos of the moon.

I'm scared that the "AI" marketing will make it much harder convincing non-technical coworkers and execs that "garbage in, garbage out" is a real concept, that not all "data" is good, and that our systems need to keep track of which kind is going where.

"All data is useful, the more the better! Just put it all into the AI and it'll sort it out."

Is this really pervasive? E.g. To my knowledge the "AI" enhancement that iPhones do automatically is limited to the usual sorts of post-processing for contrast, color, etc. There is an AI editing mode that leans more into generative fill capability that would be analogous to the Samsung incident but I don't think it's happening automatically to every photo you take.

I still remember Samsung faking images when using digital zoom...

https://www.theverge.com/2023/3/13/23637401/samsung-fake-moo...

Yes, I believe that is what the person I replied to was referring to specifically.

We faced this problem recently when doing an old-fashioned experiment to test a hypothesis. Can you trust the images from your phone to check if phenomenon X has occurred or not? What if the AI model processing the image just hallucinated what you wanted to see?

It is Descarte's evil demon incarnate. Yet another incentive to preserve my old mirrorless camera.

Nice! Just because you’re the top comment, I feel it is helpful to quick readers to point out that it has 0 to do with what happened here. :)

> They, for example, replace digits with others in scans. The replacement digits are layouted perfectly into the page, so the errors are hard to see.

I dealt with this where our fax number had a 6 in it and it would sometimes get changed into an 8, which happened to be a valid fax number for another company, ugh. And this was confidential info too…

Always a funny phone call when they insist they sent it to the number on the cover page we sent and then they send us a copy and xerox made it wrong.

> They, for example, replace digits with others in scans. The replacement digits are layouted perfectly into the page, so the errors are hard to see.

Is this with JBIG2? I remember reading about JBIG2 also used in the FORCEDENTRY zero click exploit that which was (?) used in the Pegasus spyware. Unrelated tidbit, I guess.

The Xerox problem was with image compression. The compressor would look at an incoming block, decide it was similar enough to one already noted and reuse the existing token. With small fonts and similar characters one difference was within the tolerance and after seeing something like 888 it could then decide 8B8 was the same thing. As it could only happen when things lined up perfectly the alterations would also be lined up perfectly.

Yes, this algorithm is called JBIG2.

Yes.