Benchmarking Intel, AMD and Graviton Using Erasure Coding Workloads

Re:Benchmarking Intel, AMD and Graviton Using Erasure Coding... by tecleandor on Tuesday May 23, 2023 @07:24AM (#36042754)

I'm not sure what are you referring to with "data loss protection" in ZFS, but if you refer to the 'copies' parameter, that's not erasure coding, that's just full copies of the data blocks [0]. They mostly only work for some sector errors on disk. If you lose a whole disk, THEY DO NOTHING. [1]

"The block pointer can store up to three copies of the data each pointed by a unique DVA. These blocks are referred to as “ditto” blocks in ZFS terminology." [2]

And although VMWare calls RAID5/6 "Erasure Coding" [3](!!), I'd say it's not [4].

  0: https://docs.oracle.com/en/operating-systems/solaris/oracle-solaris/11.4/manage-zfs/copies-property.html
  1: https://jrs-s.net/2016/05/02/zfs-copies-equals-n/
  2: https://pages.cs.wisc.edu/~kadav/zfs/zfsrel.pdf
  3: https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.virtualsan.doc/GUID-AD408FA8-5898-4541-9F82-FE72E6CD6227.html
  4: https://www.itprotoday.com/storage/erasure-coding-vs-raid-which-right-and-when

Edit: some extra info and links

[ Parent | on HN ]

Re:Benchmarking Intel, AMD and Graviton Using Erasure Coding... by naranha on Tuesday May 23, 2023 @07:35AM (#36042852)

Sorry I meant to write protection against data corruption (checksums)

[ Parent | on HN ]

Re:Benchmarking Intel, AMD and Graviton Using Erasure Coding... by tecleandor on Tuesday May 23, 2023 @07:52AM (#36043002)

Note that ZFS (and probably other fs) checksums are not data corruption protection, but data corruption detection ;)

If a checksum error is detected, you depend (mostly) on your RAIDZ protection to recover from it [0]. If you're on a single drive or striped array without mirrors or raidz1/2/3... you're SOL. (ditto blocks can help but they aren't the best. I'd only recommend them if you're stuck on a single drive setup without raid)

But you at least know that your data is corrupt, and that's very important too :)

  0: https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Checksums.html

[ Parent | on HN ]

Re:Benchmarking Intel, AMD and Graviton Using Erasure Coding... by matja on Tuesday May 23, 2023 @07:59AM (#36043077)

> For double or triple parity, we use a special case of Reed-Solomon coding

https://github.com/openzfs/zfs/blob/zfs-2.1-release/module/z...

[ Parent | on HN ]