News for hackers. Stuff that matters.
Re:Smaller and faster data compression with Zstandard by ctur on Wednesday August 31, 2016 @02:16PM (#12400300)
Yann will be giving a talk on Zstandard at today's @Scale 2016 conference, and the video will be posted. He can answer the most technical questions about Zstandard, but I may be able to answer some as well; we both work on compression at Facebook.
Re:Smaller and faster data compression with Zstandard by tmd83 on Wednesday August 31, 2016 @02:30PM (#12400401)
I am really looking forward to this. I usually like to read more than vidoes but for complicated topic with a good presenter it can actually be a comprehensive starting point. Would the video also be posted today or we will have to wait?

One thing I haven't figured out from either today's post or Yann's blog is whether Zstandard is switching between huff0 and FSE depending on compression level or is it somehow using both together? Also the post says its both OoO friendly and multi-core friendly but the speed benchmarks are those in a single core context or multi-core? Does only the format/algorithm multi-core friendly or the standard cli can run multi-threaded.

Re:Smaller and faster data compression with Zstandard by ctur on Wednesday August 31, 2016 @03:15PM (#12400718)
I'm not sure when videos will be posted -- hopefully soon.

All benchmarks today are single threaded. The algorithm itself is single threaded, but can be parallelized across cores. We will soon release a pzstd command line utility to demonstrate this, similar to pigz, which accelerates both compression and decompression.

Zstandard uses both huff0 and FSE together when it compresses -- it doesn't switch between them based on the input.

Re:Smaller and faster data compression with Zstandard by twitch_checksum on Monday September 05, 2016 @12:11AM (#12428051)
Re:Smaller and faster data compression with Zstandard by socmag on Thursday September 01, 2016 @06:32AM (#12404187)
Hey quick question, and sounds awesome.

If I wanted to use this in the pipeline of my servers Journaling system, is there any requirement that I restart the stream periodically.

That is to say, should I use it per journal entry (probably not a good idea for short messages), for the entire uptime of the writer, or with periodic restarts?

Obviously can measure and find out for myself, but wondered if you had any thoughts. Thanks!