News for hackers. Stuff that matters.
Re:Zfec – Efficient, portable erasure coding tool by sliken on Thursday November 17, 2016 @06:47AM (#12976563)
Anyone know of a comparison of similar tools. I've tinkered with this one: https://github.com/klauspost/reedsolomon/

I believe it's compatible with the Java implementation used by Backblaze and others.

Re:Zfec – Efficient, portable erasure coding tool by vu3rdd on Thursday November 17, 2016 @10:40AM (#12978349)
One of the authors of Tahoe-LAFS and the original creator of zfec, Zooko, co-authored a paper on the performance of such codes.

http://nisl.wayne.edu/Papers/Tech/code-pf-fast09.pdf

Luigi Rizzo is the original author of the fec C library.

I co-maintain zfec (and the Debian package of it) along with Zooko. A new release of zfec is planned to be out soonish.

edit: removing the <> delimiters around the URL. Sorry!

Re:Zfec – Efficient, portable erasure coding tool by zokier on Thursday November 17, 2016 @11:52AM (#12979060)
HN sadly does not support <> URL delimiters, so your link ends up 404. Here is it without delimiters:

http://nisl.wayne.edu/Papers/Tech/code-pf-fast09.pdf

Re:Zfec – Efficient, portable erasure coding tool by vu3rdd on Thursday November 17, 2016 @12:05PM (#12979199)
Thanks. Updated the link.
Re:Zfec – Efficient, portable erasure coding tool by jorangreef on Thursday November 17, 2016 @07:30AM (#12976836)
I don't know of any cross-library comparison benchmark. Also, if a library does ship with an internal benchmark, it usually uses a different number of data/parity shards and different shard size, compared to other library benchmarks, which makes comparison difficult.

I wrote an MIT-licensed ReedSolomon module for Node [1], with native C++ multi-core throughput. The internal benchmark (node benchmark.js) provides latency and throughput measurements for data=10, parity=4 for a variety of shard sizes, and the number of data/parity shards can be adjusted for comparison.

It's not yet using optimized assembler or SIMD like Klaus Post's implementation, but it is compatible and also based on Backblaze's implementation, as well as including a pure Javascript implementation for browser use.

[1] https://github.com/jorangreef/reed-solomon

Re:Zfec – Efficient, portable erasure coding tool by nicolast on Thursday November 17, 2016 @01:32PM (#12980026)
I ported Klaus' library to Haskell, and redid the whole SIMD implementation (see below for the extras this brings): https://github.com/NicolasT/reedsolomon/

In my library, the SSSE3 code also has AVX and AVX2 equivalents, providing higher throughput. There are also implementations with specific optimized instructions for ARM NEON, ARM64, PPC64/AltiVec and Power8. All of this in C (with intrinsics), not ASM, so making use of your C compiler's instruction scheduling, loop unrolling,... See https://github.com/NicolasT/reedsolomon/blob/master/cbits/re...

Re:Zfec – Efficient, portable erasure coding tool by nickcw on Thursday November 17, 2016 @08:39AM (#12977277)
The one you linked to is excellent. It has hand crafted SSE3 assembler for enormous speed. It is a bit difficult to compare the benchmarks from the one in the original posting to Klaus's as they aren't done at the same (N,K) parameters though.
Re:Zfec – Efficient, portable erasure coding tool by ironMann on Thursday November 17, 2016 @12:19PM (#12979345)
Papers usually compare against jerasure: http://jerasure.org/jerasure/jerasure