Why are tar.xz files 15x smaller when using Python's tar compared to macOS tar?

Re:Why are tar.xz files 15x smaller when using Python's tar ... by derefr on Sunday March 14, 2021 @07:54PM (#26460845)

I'm not an information theorist / cryptographer / etc., and so I have a strong feeling that there must be something wrong with my assumptions, given that "compressing big files using lots of memory for people to later decompress back to files using average amounts of memory" is an extremely-common need (for e.g. packaged software/OS updates.)

When a physicist sees a proof for a perpetual-motion machine, they don't ask "can this work"; they ask "where is this going wrong, that it came to this absurd conclusion?"

My post wasn't suggesting a real technique — it was me sharing my own perpetual-motion machine proof, so that someone can hopefully point out the step that's impossible, and I can then learn something from that :)

[ Parent | on HN ]

Re:Why are tar.xz files 15x smaller when using Python's tar ... by dmurray on Sunday March 14, 2021 @08:27PM (#26461135)

It sounds plausible to me. Real-world constraints aren't usually "you can use any amount of resources on the compressor, and any amount of compute on the decompressor for free, but the decompressor is hardcapped at exactly X MB of memory". So you might only have a theoretical improvement on existing methods, but that would still be very cool.

You can't beat existing methods by much - we're pretty close to the Shannon limit - but it's not a closed subject.

[ Parent | on HN ]