1. This is obviously a (funny) joke (pretty great article stav!)

2. Any image generated with stable diffusion, could be distributed as prompt + seed leading to 99% compression

3. Image -> downsample –> super-resolution model -> render -- this must already exist as a compression model? The only limitation is that super-res models probably can't run on phones yet... but some day soon they'll be good enough.

4. All deep learning is fundamentally a compression system (reduce a lot of input data to a set of finite features and weights). But it might also be a very good lossy compression system because it can identify what's worth keeping high res and what's worth aggressively downsampling.

5. Interestingly, we seem to be compute limited more than bandwidth limited nowadays so it might take a while for an economically useful DL-compression product to emerge.

Reminds me of the old IBM joke: you can compress everything down to a bit, if you include the data in the decompresser.

> this must already exist as a compression model?

Definitely exist in video compression, not so sure about still image (but most of still image compression advancements lately are just side effect of video compression).

I wonder how close to an arbitrary image a truly optimal Stable Diffusion prompt could actually get (the ones on the blog being just a good guess). Like say you magically could try infinite prompt variations to find the closest match. Maybe some of the text would look like encoded gibberish. I imagine the "optimal" prompt would be very long, maybe - probably - larger than the original image.

bonus points for Getty watermark in one of the "results"

I've seen other Stable Diffusion examples where it starts with a low quality image and a prompt, and makes a higher-quality version.

I'd be really interested to see this work extended with something that transmits (say) a 64x64 pixel copy of the original image and uses it as source material for Stable Diffusion to work on.

Depending on the parameters it might be a legitimately viable image compression algorithm, at least for some types of images.

Hilarious post aside, I've been using Stable Diffusion to generate "ideas" in large volume by creating a lot of images from a single (very generic) prompt with very low accuracy/depth levels, then when I find something that might be interesting, I fire up img2img and iterate over that with better accuracy, resolution, and depth. The results are pretty amazing.

That's a great idea!

I've been curious, isn't img2img basically just more SD passes? Ie you generate something with 5 passes, then img2img just runs the other 95 over that to "complete" the image.

Basically, as far as I understand it (which is not much), txt2img generates a very rough blob from a prompt, then img2img does additional passes over that to refine it to something that has much higher quality. Is this incorrect?

I'm actually not sure how it works specifically because I don't know much about the technology behind it. All I know is that if I change the prompt (compared to the original prompt given by the txt2img script) and finetune the parameters I can get some *very* differing results that only keep a certain "style" or vibe.

For example I generated [0] as my base image, I passed it into a dithering filter to make it look more "retro" and got [1] (was going for retro PC98 visual novel style backgrounds), and then [3] are all new results generated from it (I added stuff like "village" or "trees" etc in the new prompt and also made the resolution widescreen). You can tweak parameters like the prompt, the depth (ddim), the "strength" of the original image, and the "scale" (not sure what it is but it does change things around). Then you can play with that and even iterate on more images.

[0] https://upload.i.ng/file/goaltendersgrievousSumter/seed_6436...

[1] https://upload.i.ng/file/overstatementrearwardsplighting/dit...

[2] https://twitter.com/xMorgawr/status/1564271156462440448

This technique needs to be adopted everywhere immediately. With infinite gains over jpeg, how could we delay?

The commentary gave me a good chuckle, but I was actually pretty impressed with some of the results. I wonder if using stock images is playing to the strength of the model, because they may have actually been part of training.

I think that's actually the case, yeah. I think it mostly "remembered" these images, as it had been trained on them.

I'm _pretty sure_ this is tongue-in-cheek: the final "uncompressed" image bears only a superficial resemblance to the original.

> There are some minor kinks that need to be worked out, such as the fact that each image takes around a day to generate on mobile, but this is more than acceptable in certain domains. Website visitors, for example, are well-accustomed to such loading times, and would barely notice any difference.

You're right but I can see a future where an image can be analyzed by an AI and get the minimum amount of data to recreate it.

Obvious application is situations extremely constrained by bandwidth. A space explorer's family could transmit lovingly promptcrafted pictures to a station on a distant planet.

(Videos should similarly lossy-compress down to a plot synopsis and the key cast/crew. Throw in the script too for more accuracy but biggger file.)

Pictures alone are pretty low bandwidth with good compression. While the concept sounds cool, I don't see any immediate application for it. For long distance communication (/interplanetary) it's a problem of latency, not bandwidth. However, transmitting moderate/large length high-res video clips from prompts could be interesting, probably magnitude more efficient than existing compression algorithms.

“quality is better than the original, and it is astonishing for a compressor to achieve 100%+ quality.”

Wow. Sounds like we’ve got a new standard image format on the the way. With a continuous “truthy” scale rating.

believe half of what you see and nothing of what you hear...

We may have AI-powered "reality compression" in the next five years when mixed reality goggles/glasses become more comfortable and popular as mainstream alternatives for actually physically visiting places and people.

Which will basically create realistic 3d streams of dynamic environments and people (so they can be viewed stereoscopically and from different perspectives).

The enhanced AI-based compression should make it more practical to stream in the moving 3d data with fewer delays and latency and more realism than something like a traditional game engine.

Unless no one wants to go to real places anymore because imaginary ones are nicer.

But think about virtual reality with realistic faces and eye contact.

Or a movie where you can view a scene from any perspective you want at any time. Even one of the protagonists. When you are trying to realistically transmit and recreate many entire dynamic 3d scenes, that is where AI can help a lot with scanning and compression/decompression.

Ya. If our universe is a simulation, perhaps our perceived reality is generated using something like stable diffusion.

Ages ago, for some raytracing stuff, a buddy used l-systems to generate forests, gardens, etc. Each plant was generated from a seed. A very compact reproducible, representation.

https://en.wikipedia.org/wiki/L-system

https://en.wikipedia.org/wiki/The_Algorithmic_Beauty_of_Plan...

The compactness of L-systems makes me think it might be a good idea to combine these more traditional types of procedural generation with AI-based generation to take advantage of the efficiency. Especially if we were trying to quickly load in something like a virtual world (rather than needing to replicate a real one). The L-systems rules are probably much more compact than the AI vectors. And there might even be some parallel versions.

What we know about our brains (whether it's a simulated brain or not) tells us that effectively our perception is a simulation of our own. Because for example as you look around, your eyes only get high resolution data in the center, and that can only represent a fraction of the relevant immediate environment. And none of the senses or motor activations are quite real time, there is always some type of lag. The brain takes the data samples and uses them to fine tune a generative prediction system that constructs the dynamic and holistic 3d scene and your position in it using the computational power of your own network of neurons.

To me, dreams provide a certain level of proof that we can do this type of simulation, and something similar is happening while we are awake, but with environmental data sampling and other cognitive processes feeding in.

Jokes aside, that the 250 terabytes of the LAION 5B dataset are remotely able to be, in any form, represented by the weights contained within the sub 5-gigabyte file that the represents StableDiffusion 1.4 model download, is nothing short of amazing.

This technology might actually make The Stick of Jan Sloot [0], which I learned about on here recently, possible.

The idea behind the stick was that you'd have a decoder next to your TV that would contain all the movies ever made encoded in a special format, and if you wanted to watch a particular one, you just needed a specific key that told the decoder how to decompress it.

It didn't really work, as you'd imagine.

[0] https://spronck.net/sloot.html

I think much like the stick, or a Minecraft seed, or other examples of procedural generation, you can get a near infinite amount of results via different seeds from a program with a relatively small total size. But you cannot ever get an exact result you want, unless it's already one of the outputs of the algorithm.

Much like the key described in your article, it may be possible to get a very accurate result from an AI via a very long and specific prompt. But that long prompt is then where the "magic" compression goes away.

Pretty much the old trick of moving the bulk of the complexity into the compressor/decompressor, including the content itself, and taking that into the AI realm.

You can always reproduce the exact image by using the same prompt, seed value, and config info (assuming you've created the image with SD to start with).

Is there a unique image identifier system that might facilitate the preservation of this metadata?

You would think but there seems to be some kind of randomization added to the image creation. You can use the same prompt over and over again and get different images.

When I use the same seed and same prompt, I get the same image.

Same seed, same prompt and same hardware perhaps, if the GPU concurrency doesn't introduce nondeterminism.

Otherwise you might have the same problem of this: https://github.com/EmbarkStudios/texture-synthesis#notes

> When using multiple threads for generation, the output image is not guaranteed to be deterministic with the same inputs. To have 100% determinism, you must use a thread count of one, which can by done via

> CLI - texture-synthesis --threads 1

> API - SessionBuilder::max_thread_count(1)

Compression algorithms using floating point have always been flaky though. Need a fixed-point image model.

You are correct. My mistake.

I've played a lot with this and I get the same image using the same prompt and seed.

I've also played a lot with slight variations on the same prompt and it's really fascinating seeing how similar the images are outside of the specific detail changed.

I would say this diffusion model is very... stable.

Use stable diffusion for image compression

Store stable diffusion model in the π file system (https://github.com/philipl/pifs)

"Almost zero overhead in terms of data storage"

I wonder if iterated function systems used for fractal image compression could be combined with AI to make a similar image compression method that actually works:

https://ieeexplore.ieee.org/document/918789

https://link.springer.com/chapter/10.1007/978-0-387-69216-6_...

Diffusion models can be used to lossily compress images for real, see for example Lossy Compression with Gaussian Diffusion (https://arxiv.org/abs/2206.08889). I would expect this to work with text-conditional diffusion models too and give you better compression when supplying a correct caption.

This is better than middle-out

I'm not sure compression with deep learning makes sense though? Currently data storage and transportation costs are much cheaper than computation. Until that's reversed, I'm not sure this is really a viable use case? And just intuitively, since both costs are proportionate to the size of the raw data, I don't see that happening with deep learning models.

You still have to transfer the generated image from stable diffusion to your browser... How do you compress that? :-D

You don't, you generate it in the browser.

To nitpick a joke, jpeg cant have infinite size, since the format only supports at most 2^16 × 2^16 pixels.

Well you can put anything into APPn segments, at most 65,536 bytes each, unlimited times.

I believe that video codecs will probably work along these lines at some point. Maybe have a base AI layer and then a more traditional codec layer on top to refine it.

I think there are already AI-based codecs for reducing bandwidth of video calls with faces.

The JPEG-XR link probably was meant to point to https://en.m.wikipedia.org/wiki/JPEG_XL instead.

Oops, will fix, thanks!

What's going on with that deformed bowl in "three bowls of food on a white table" one?

Also, "a couple of women"? Is it part of the joke or the right one is really a woman?

No, that's just what img2prompt thought.

Okay, I've been really negative about AI image generation, but this is a really cool use for it.

So reading the comments, this answers my question if this was a joke.

... games that come on multiple floppy disks! :-D

This is NOT compression. Almost all information gets lost. This is simply about replacing images with captions + image generator.

Probably shouldn't have to say it but, that's the joke.

Image to Prompt is discussed in this article. I’m scared to find out what adjectives it would use to prompt my bio picture.. https://replicate.com/methexis-inc/img2prompt