News for hackers. Stuff that matters.
Re:Smaller and faster data compression with Zstandard by Twirrim on Thursday September 01, 2016 @01:30AM (#12403281)
From the bits of testing I've done today, it's phenomenally fast on x86. Much better than gzip (and pigz for that matter) in every metric I think I generally care about: CPU Usage, Compression Speed, Decompression Speed, Compression Ratio.

On other architecture the picture gets a bit murky, it seems to get handily beaten by pigz through what at first blush I'd guess is just sheer parallelism. It's got solid performance, and without a shadow of doubt faster than vanilla gzip. If/as/when I get time, it'll be interesting to dig into why performance is worse there.

Re:Smaller and faster data compression with Zstandard by Twirrim on Thursday September 01, 2016 @11:26AM (#12406004)
Dug in a bit further. On the non-x86 architecture I use, it looks like it's really just straight core performance that explains it. pigz's only advantage there really seems to be the brute force parallelism.

In particular note the huge difference in branches between gzip and zstd on decompress:

8959780663 branches # 143.024 M/sec

2969481781 branches # 64.454 M/sec

and on misses:

542158823 branch-misses # 6.05% of all branches

89060880 branch-misses # 3.00% of all branches

Re:Smaller and faster data compression with Zstandard by Cyan4973 on Thursday September 01, 2016 @10:20PM (#12410383)
There is now a pzstd implementation, if you wish to compare it to pigz
Re:Smaller and faster data compression with Zstandard by Twirrim on Friday September 02, 2016 @01:54PM (#12414842)
Had a shot. It's slightly buggy, but holy crap is it fast.

I'm not a C programmer, understanding what happened is a bit beyond me but: 1) to compile on linux it needs the -pthread flag passed to it, Makefile is missing that (compiles fine on OS X) 2) decompression over stdin appears to be effectively impossible, still demands in input file. Compression over stdin works fine.