Is your NAS NVME based if you are using spinning metal you won't reach speeds to saturate 10GbE especially with small file sizes?

I have a Linux NAS with 6 7200RPM HDDs in a raidz2 ZFS pool, and the local fs read speed on large files is 7 times faster than GbE, about ~930 MB/s. So while this wouldn't saturate a 10GbE link, it would still be greatly beneficial to multiply the network read speed by 7-fold by upgrading from GbE to 10GbE.

How did you bench your ZFS pool?

With dd. And normally when I do this I drop the caches with "echo 3 > /proc/sys/vm/drop_caches" but this doesn't work with ZFSOnLinux as it has a separate ARC cache that doesn't get cleared by that command (and I don't believe it's possible to clear it, other than by exporting and reimporting the zpool.)

So I read a 10GB chunk of a file that I know is not cached:

  $ dd if=/some/large/file of=/dev/null bs=1M count=10240
  10240+0 records in
  10240+0 records out
  10737418240 bytes (11 GB, 10 GiB) copied, 11.3378 s, 947 MB/s

The 947 MB/s figure printed by dd matches the speed reported by a "zpool iostat tank 1" running during the benchmarking, which confirms I'm not reading from the cache. When I repeat the same command, dd completes almost immediately reporting a speed of about 6-8 GB/s on my machine, while the zpool iostat command shows zero read bandwidth.

It's a ZFS-based NAS with an SSD cache. Right now it's only 1x SATA3 SSD which means it will top at 500MB/s (so slightly less than a half of 10GbE) and I am planning to add one more which will make it almost saturate a 10GbE link.

Additionally yep, I plan to have one NVMe dataset sometime this year.

And even further, I plan to add a few enterprise HDDs in stripe and mirror. If you have 8 (with capacity of 4 for full mirroring) and each is at 260MB/s then et voila, you get 10GbE again.

If somebody knows why they should pay a premium for 10GbE, they also know how bad spinning rust is.

Those same people should also know that spinning rust can go very fast as well... My 8 and 12 drive spinning rust pools can read at 500MB/sec easily and get close to GB/sec.

But the 10Gbit network is there for the flash pool.

Yeah, but... The noise! I just can't handle spinning rust anymore for this alone even if the $/GB ratio is still so much better.

Consumer SATA SSDs are plentiful enough that I'm just going all in on flash when I revamp my server and do an 8 drive (+1 hotswap) RAID10 build or whatever. I don't need +60TB or anything and 8-9 drives is about the limit you can find on high end consumer mobos I think...

> The noise

It is not required for the NAS to be in the same room where you sleep.

With a proper drives (and maybe some APM tinkering) you can make a very quiet one. Also using 2.5" drives is an option too.

Though if you can throw money at the problem than of course you can just buy a bunch of TB SSD drives.

Or say 1Gbps is too low bar. Even single rust drive can fill 1GbE in simple workload.

Put 8 high-quality HDDs in a stripe & mirror config. Thus you can read from 4 simultaneously. They each top at 260MB/s. That amounts to little over 1GB/s -- which is 10GbE.