From the README
> The generator is trained to achieve realistic and not exact reconstruction. It may synthesize certain portions of a given image to remove artifacts associated with lossy compression. Therefore, in theory images which are compressed and decoded may be arbitrarily different from the input. This precludes usage for sensitive applications. An important caveat from the authors is reproduced here:
> "Therefore, we emphasize that our method is not suitable for sensitive image contents, such as, e.g., storing medical images, or important documents."
As an example of this going wrong previously, xerox had once implemented compression based on deduplicating duplicate parts of documents. Obviously numbers contains tons of duplicate symbols (digits). The problem was that the scanner software deduplicated different numbers with each other, leading to wrong numbers.
http://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_...
Yes, I understand this is a lossy compression method - what I was proposing is to have the compressor as a final pass take the predicted output image, and subtract it from the original pixels. This gives you a delta between the predicted image and the original image. You can then compress that delta losslessly, and store it alongside the output of this model - if the predicted image is close enough to the original image then you've significantly reduced the amount of entropy in the delta, making it highly compressible.
This is how some domain-specific lossless compression algorithms work, e.g. DTS-HD Master Audio
However, the model does learn a conditional probability distribution over a lower-dimensional representation of the original image - this is unavoidable as entropy coding requires a distribution over discrete symbols. The GAN is almost auxiliary and not a central component of the model - in fact, you can get very good results without the GAN, but does seem to result in visually superior reconstructions.
The model in the demo is a lossy compression method because it first projects the input to a lower dimensional space and performs quantization of this representation to integer values so the result can be ultimately entropy coded. It uses the mean-scale hyperprior model introduced in [1] to estimate the necessary probability distributions in the lower-dimensional space for entropy coding.
[1]: https://arxiv.org/abs/1811.12817 [2]: https://arxiv.org/abs/1802.01436
> [4]: Sample reconstructions
The text in the reconstructed image in the third row looks different, the word phonomat is quite garbled, information looks a bit funny.
I assume most people just want to see the images, forcing them to recompute them is a waste of resources. Even just storing a version of the colab with the results present would help a lot.
The free tier for traffic is "15GB of Data Transfer Out", after that it's 9 cents per GB. https://aws.amazon.com/s3/pricing/?nc1=h_ls (under "Data transfer"). Check your AWS bill!
One interesting failure model is that images dominated by high-frequency detail require a relatively large bitrate to store - see e.g. the last example in the Github README with the weird brickwork. Even though the model was trained to produce compressed representations with a soft constraint on the maximum bitrate, the filesize of the representation for this particular image is something like 60% above the nominal maximum.
This project is based on the paper "High-Fidelity Image Compression" by Mentzer et. al. [3] - this was one of the most interesting papers I've read this year! The model is capable of compressing images of arbitrary size and resolution to bitrates competitive with state-of-the-art compression methods while maintaining a very high perceptual quality. At a high-level, the model jointly trains an autoencoding architecture together with a GAN-like component to encourage faithful reconstructions, combined with a hierarchical probability model to perform the entropy coding.
What's interesting is that the model avoids compression artifacts associated with standard image codecs by subsampling high-frequency detail in the image while preserving the global features of the image very well - for example, the model learns to sacrifice faithful reconstruction of e.g. faces and writing and use these 'bits' in other places to keep the overall bitrate low.
The overall model is around 700MB - so transmitting the model wouldn't be particularly feasible, and the idea is that both the sender and receiver have access to the model, and can transmit the compressed messages between themselves.
If you have any questions or notice something weird I'd be more than happy to address them.
---
[1] Colab Demo: https://colab.research.google.com/github/Justin-Tan/high-fid...
[2]: Github: https://github.com/Justin-Tan/high-fidelity-generative-compr...
[3]: Original paper: https://hific.github.io/
[4]: Sample reconstructions: https://github.com/Justin-Tan/high-fidelity-generative-compr...