Add 10 GbE to your system with an M.2 2280 module

Re:Add 10 GbE to your system with an M.2 2280 module by mrb on Wednesday January 19, 2022 @12:55PM (#29997274)

I have a Linux NAS with 6 7200RPM HDDs in a raidz2 ZFS pool, and the local fs read speed on large files is 7 times faster than GbE, about ~930 MB/s. So while this wouldn't saturate a 10GbE link, it would still be greatly beneficial to multiply the network read speed by 7-fold by upgrading from GbE to 10GbE.

[ Parent | on HN ]

Re:Add 10 GbE to your system with an M.2 2280 module by csdvrx on Wednesday January 19, 2022 @01:27PM (#29997831)

How did you bench your ZFS pool?

[ Parent | on HN ]

Re:Add 10 GbE to your system with an M.2 2280 module by mrb on Wednesday January 19, 2022 @04:49PM (#30000723)

With dd. And normally when I do this I drop the caches with "echo 3 > /proc/sys/vm/drop_caches" but this doesn't work with ZFSOnLinux as it has a separate ARC cache that doesn't get cleared by that command (and I don't believe it's possible to clear it, other than by exporting and reimporting the zpool.)

So I read a 10GB chunk of a file that I know is not cached:

  $ dd if=/some/large/file of=/dev/null bs=1M count=10240
  10240+0 records in
  10240+0 records out
  10737418240 bytes (11 GB, 10 GiB) copied, 11.3378 s, 947 MB/s

The 947 MB/s figure printed by dd matches the speed reported by a "zpool iostat tank 1" running during the benchmarking, which confirms I'm not reading from the cache. When I repeat the same command, dd completes almost immediately reporting a speed of about 6-8 GB/s on my machine, while the zpool iostat command shows zero read bandwidth.

[ Parent | on HN ]