News for hackers. Stuff that matters.

mpais

Re:Handsum: An LQIP Image File Format on Sunday July 12, 2026 @09:55AM (#48881274)
Shameless plug, SQZ does just that, you can check it out at https://github.com/MarcioPais/SQZ

Also for some discussion on more technical aspects and sample images, check out the thread over at the Encode Forum: https://encode.su/threads/4183-SQZ-Low-complexity-scalable-l...

> What environments do you see this benefiting?

Anywhere where you have visual content that needs to be "consumed" in a large variety of possible viewings conditions, from low to extremelly high DPI 5 or 6-inch smartphone screens, to 10-inch tablets, to 60-inch TVs, and all at possibly variable quality.

Suppose your application uses a carousel-like UI for viewing photographs, so you need small (say scaled with correct aspect ratio to fit in 128px by 128px) thumbnails for the scrollable list at the bottom, a preview (1024px) for the center of the UI, and then the actual full size image for when the user clicks the preview. Codecs designed for progressive transmission allow you to use just one high-quality version of each image and obtain lower quality versions of those by simply sending only part of that file, or, conversely, allow you to refine the preview you've already received by simply sending more bits from the complete file, without the need to retransmit everything again. So you don't need 3 or however many versions of the same image encoded in different files with a format that doesn't have this scalability.

> However, what’s the overhead for this? How well does this “encode once, read many” style scale compare to the typical “encode many, read many” in terms of server resources?

That's the thing, you reduce the overhead with these scalable formats, either by reducing storage cost by not having to store multiple files per image, or computational costs if you planned on just transcoding on-the-fly to a lower quality and serving that, to a bit of both as you'd expect from a CDN that caches transcoded versions.

This isn't to say that such scalable formats are without downsides. The server still needs to know how much of the full quality image to serve based on the requested client needs, but this is usually trivial and of much less computational costs. The main questions are always if the compression ratio is heavily impacted or not, and whether the perceived visual quality remains acceptable when extracting heavily compressed representations for downscaled viewing, i.e., what is the useful range of this scalability. That is why I tuned SQZ to attempt to preserve as much detail as possible early on in its bitstream, so that when downscaling you still get good looking images. No downscaler is magically going to recover the details that were blured out by the heavy filtering that modern codecs do.

If you check out the discussion in the forum I linked in the description (don't be scared by its TLD, it's a great resource in this field, an old "hidden gem" in today's frivolous web) you'll find example images and analysis describing such questions.

And please bear in mind that SQZ is just a little fun thought experiment that I decided to implement, it has no grandiose ambitions. But if you need something like this for a quick side project, and don't want to use a 250k LOC library with who knows how many dependencies, then maybe try it out and let me know what you think.

The codec is extremely simple (hence why it's relatively fast even with such unoptimized code) and as such doesn't even use any entropy coding backend, which is what allows for the incredible scalability. Now, obviously, that means that when going for high quality output, it can't really match JPEG-XL, but that's just a tradeoff - using context-modelling would allow it to beat JXL, but then you couldn't really have byte-granularity scalability. So you can tell it to match the exact same file size of your JXL image, and compare both.

At nearly perceptually lossless quality levels, even if it doesn't match JXL, it usually is quite acceptable also, and at extremelly high compression ratios (think over 250:1) it's usually better than JXL, which is important, otherwise the scalability would be a moot point.

It was designed with a particular use case in mind - simplifying the process of serving visual content in a responsive environment. Instead of encoding the same image at 4 or 5 different resolutions and then serving the appropriate one based on the requested specifications - which usually would mean either storing those multiple versions or reencoding on-the-fly on an as-needed basis - you encode the image only once, at a quality you deem matches your maximum requirement; it can even be completely lossless.

Then when you receive a request for an image at a lower quality/resolution, you can simply send only the first 1/10 of the file, or 1/8, or however you like. You can literally just truncate the file at a random offset and when you decode it, you just get a lower quality version of the image, but which is still exactly the best the codec could have done at that file size anyway. The requesting party can then simply downscale the lower quality image, hence why it's important to have good details even at high compression ratios, so that the downscaled version will look good. The format allows for all that, even progressive enhancement, but the provided library for now doesn't support all of this.

There are other formats with similar scalability, but they're usually a lot more complex - SQZ is basically just a fancy run-length codec, in a single file header for C/C++.

Re:Compressing Text into Images on Sunday January 14, 2024 @02:25PM (#38993558)
> Lossless ones are basically just ZIP

Lossless image compression is a vast field with myriad different requirements and solutions, and generalizing like that is just plain wrong. Most lossless image codecs for photographic or similar rendered content don't use a Lempel-Ziv stage like PNG does, they exploit the spatial and possibly spectral correlations to form predictions, be it with simple linear predictors or more complex approaches. However, for simple non-photographic images like cartoons or some such, the LZ-based stage can provide significantly better compression than the simpler prediction-based methods.

Even if the premise of the post is flawed, it's always nice to see people interested in data compression and trying to learn more about it, so I'm all for more "nonsense" posts if it helps draw attention to the field.

Show HN: SQZ – Low complexity, scalable lossy to lossless image compression

https://github.com/MarcioPais/SQZ

I'd like to share a little toy project of mine, a really simple image codec that can do lossy to full lossless image compression with complete scalability at a byte level granularity - you can compress an image just once, even fully losslessly if needed, and then get any lossy version possible by simply stopping decompression at any offset in the compressed data.This "encode onde,...

( Read More... | 4 comments | 6 points | on HN )
Re:QOI – The Quite OK Image Format on Saturday April 02, 2022 @02:51AM (#30886337)
QOI is not on the pareto frontier for compression [1], it is however extremely simple and is a nice introduction to data compression for anyone wanting to learn about the field.

[1] See:

https://www.lucaversari.it/FJXL_and_FPNGE.pdf

https://github.com/libjxl/libjxl/tree/main/experimental/fast...

https://github.com/veluca93/fpnge

Re:Dictionary Compression on Tuesday March 01, 2022 @03:37PM (#30519654)
paq8px v206 [1] with option "-8art" compresses it to 9,145 bytes using ~2GB of memory in 82s, mostly due to a better pre-training stage.

[1] https://github.com/hxim/paq8px

Re:Universal algorithm set to boost microscopes on Friday August 30, 2019 @12:57PM (#20840334)
Some RAW formats are pre-compressed, usually with very simple (hence fast) algorithms, and some aren't even lossless (Sony ARW). It's often possible to further compress them with specialized tools.

Disclosure: I'm the author of PackRAW, a tool that does exactly that (https://encode.su/threads/2762-PackRAW)

Re:NNCP: Lossless Data Compression with Neural Networks on Saturday April 06, 2019 @11:32AM (#19591218)
"his relatively simple LSTM-based neural net compressor is 10x faster than one of the more complex state of the art algorithms while delivering comparable compression ratios."

No, it's 10x faster than an experimental, completely performance-unoptimized LSTM model that is far from state-of-the-art on its own. From the paper:

"Regarding the speed, our results show that our implementation is much faster than lstm-compress[8] with a similar model and gain. The speed improvement comes from the more optimized matrix multiplication implementation and the larger batch size (16 instead of 1)."

Of most interest here is the library used, LibNC, which might allow us to speed up cmix a bit (even making it 2x faster would already help a lot in testing).

Disclaimer: I've worked on cmix with its author