I won't be bothered in a home network.
> Clonezilla are vastly more moving parts
...and one of these moving parts is image integrity and write integrity verification, allowing byte-by-byte integrity during imaging and after write.
> I strongly recommend against oflag=direct as in this... [snipped for brevity]
Unless you're getting a bottom of the barrel NVMe, all of them have DRAM caches and do their own write caching independent of O_DIRECT, which only bypasses OS caches. Unless the pipe you have has higher throughput than your drive, caching in the storage device's controller ensures optimal write speeds.
I can hit theoretical maximum write speeds of all my SSDs (internal or external) with O_DIRECT. When the pipe is fatter or the device can't sustain that speeds, things go south, but this is why we have knobs.
When you don't use O_DIRECT in these cases, you see initial speed surge maybe, but total time doesn't reduce.
TL;DR: When you're getting your data at 100MBps at most, using O_DIRECT on an SSD with 1GBps write speeds doesn't affect anything. You're not saturating anything on the pipe.
Just did a small test:
dd if=/dev/zero of=test.file bs=1024kB count=3072 oflag=direct status=progress
2821120000 bytes (2.8 GB, 2.6 GiB) copied, 7 s, 403 MB/s
3072+0 records in
3072+0 records out
3145728000 bytes (3.1 GB, 2.9 GiB) copied, 7.79274 s, 404 MB/s
If the drive didn't have its own cache, caching on the OS side would have more profound effect since I can queue more writes to device and pool them at RAM.
This matters in the specific use case of "netcat | gunzip | dd" as the compressed data rate on GigE will indeed be around 120 MB/s but when gunzip is decompressing unused parts of the filesystem (which compress very well), it will attempt to write 1+ GB/s or more to the pipe to dd and it would not be able to keep up with O_DIRECT.
Another thing you are doing wrong: benchmarking with /dev/zero. Many NVMe do transparent compression so writing zeroes is faster than writing random data and thus not a realistic benchmark.
PS: to clarify I am very well aware that not using O_DIRECT gives the impression initial writes are faster as they just fill the buffer cache. I am taking about sustained I/O performance over minutes as measured with, for example, iostat. You are talking to someone who has been doing Linux sysadmin and perf optimizations for 25 years :)
PPS: verifying data integrity is easy with the dd solution. I usually run "sha1sum /dev/nvme0nX" on both source and destination.
PPPS: I don't think Clonezilla is even capable of doing something similar (copying a remote disk to local disk without storing an intermediate disk image).
I noted that the bus I connected the device has 500MBps bandwidth theoretical, no?
To cite myself:
> Target is a Samsung T7 Shield 2TB, with 1050MB/sec sustained write speed. Bus is USB 3.0 with 500MBps top speed (so I can go %50 of drive speeds). Result is 404MBps, which is fair for the bus.
The 5Gbps ports are just marketed as "USB 3.1" instead of "USB 3.0" these days, because USB naming is confusing and the important part is the "gen x".
USB 3.0, USB 3.1 gen 1, and USB 3.2 gen 1x1 are all names for the same thing, the 5Gbps speed.
USB 3.1 gen 2 and USB 3.2 gen 2x1 are both names for the same thing, the 10Gbps speed.
USB 3.2 gen 2x2 is the 20Gbps speed.
The 3.0 / 3.1 / 3.2 are the version number of the USB specification. The 3.0 version only defined the 5Gbps speed. The 3.1 version added a 10Gbps speed, called it gen 2, and renamed the previous 5Gbps speed to gen 1. The 3.2 version added a new 20Gbps speed, called it gen 2x2, and renamed the previous 5Gbps speed to gen 1x1 and the previous 10Gbps speed to gen 2x1.
There's also a 3.2 gen 1x2 10Gbps speed but I've never seen it used. The 3.2 gen 1x1 is so ubiqitous that it's also referred to as just "3.2 gen 1".
And none of this is to be confused with type A vs type C ports. 3.2 gen 1x1 and 3.2 gen 2x1 can be carried by type A ports, but not 3.2 gen 2x2. 3.2 gen 1x1 and 3.2 gen 2x1 and 3.2 gen 2x2 can all be carried by type C ports.
Lastly, because 3.0 and 3.1 spec versions only introduced one new speed each and because 3.2 gen 2x2 is type C-only, it's possible that a port labeled "3.1" is 3.2 gen 1x1, a type A port labeled "3.2" is 3.2 gen2x1, and a type C port labeled "3.2" is 3.2 gen 2x2. But you will have to check the manual / actual negotiation at runtime to be sure.
It's not intended to be used by-design. Basically, it's a fallback for when a gen2x2 link fails to operate at 20Gbps speeds.
Fun fact: "openssl sha1" on a typical x86-64 machine is actually about twice faster than "sha1sum" because their code is more optimized.
Another reason I don't bother to use tee to compute the hash in parallel is that it writes with a pretty small block size by default (8 kB) so for best performance you don't want to pass /dev/nvme0nX as the argument to tee, instead you would want to use fancy >(...) shell syntax to pass a file descriptor as an argument to tee which is sha1sum's stdin, then pipe the data to dd to give it the opportunity to buffer writes in 1MB block to the nvme disk:
$ nc -l -p 1234 | tee >(sha1sum >s.txt) | dd bs=1M of=/dev/XXX
Instead of the "fancy" syntax I used
mkfifo /tmp/cksum
sha1sum /tmp/cksum &
some_reader | tee /tmp/cksum | some_writer
Disclaimer: From memory and untested now. Not.at the keyboard.
dd followed by sha1sum on each end is still very few moving parts and should still be quite fast.
I strongly recommend against oflag=direct as in this specific use case it will always degrade performance. Read the O_DIRECT section in open(2). Or try it. Basically using oflag=direct locks the buffer so dd will have to wait for the block to be written by the kernel to disk until it can start reading the data again to fill the buffer with the next block, thereby reducing performance.