Re:A little SSH file copy magic at the command line
by wang_li on Thursday October 17, 2019 @03:42PM
(#21285653)
That doesn't seem to be the case. A relatively fast machine can't gzip anywhere near 120 MB/s unless the data ultra compressible. Zstd is multithreaded, but single threaded it's speed isn't that fast either.
$ lscpu | grep name
Model name: Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz
$ time dd if=/dev/zero of=/dev/null bs=10240 count=102400
102400+0 records in
102400+0 records out
1048576000 bytes (1.0 GB) copied, 0.145168 s, 7.2 GB/s
real 0m0.147s
user 0m0.036s
sys 0m0.111s
$ time dd if=/dev/zero bs=10240 count=102400 | gzip > /dev/null
102400+0 records in
102400+0 records out
1048576000 bytes (1.0 GB) copied, 5.36061 s, 196 MB/s
real 0m5.364s
user 0m5.317s
sys 0m0.472s
$ time dd if=/dev/urandom bs=10240 count=102400 | gzip > /dev/null
102400+0 records in
102400+0 records out
1048576000 bytes (1.0 GB) copied, 27.3478 s, 38.3 MB/s
real 0m27.352s
user 0m27.169s
sys 0m10.119s
$ time dd if=biggestwords bs=10240 count=102400 | gzip > /dev/null
10642+1 records in
10642+1 records out
108981378 bytes (109 MB) copied, 6.66068 s, 16.4 MB/s
real 0m6.669s
user 0m6.667s
sys 0m0.249s
$ time dd if=biggestwords bs=10240 count=102400 | gzip -1 > /dev/null
10642+1 records in
10642+1 records out
108981378 bytes (109 MB) copied, 1.53539 s, 71.0 MB/s
real 0m1.540s
user 0m1.522s
sys 0m0.138s
$ time dd if=biggestwords of=/dev/null bs=10240 count=102400
10642+1 records in
10642+1 records out
108981378 bytes (109 MB) copied, 0.0407698 s, 2.7 GB/s
real 0m0.043s
user 0m0.007s
sys 0m0.036s
Biggestwords is simply /usr/share/dict/words concatenated to itself 22 times so as to provide ~100 MB of data.
If you can get good enough compression it might still be worthwhile to compress. But it's pretty rare in my experience.
If you can get good enough compression it might still be worthwhile to compress. But it's pretty rare in my experience.