A little SSH file copy magic at the command line

Re:A little SSH file copy magic at the command line by wang_li on Thursday October 10, 2019 @05:02PM (#21219140)

Unless you can compress at a higher throughout than your network speed, don’t do this. I’m not familiar with too many machines that can gzip at greater than 120 Mbytes/sec, which is the network bandwidth of a switched gigabit network. There’s no point in making your task take longer with unnecessary compression steps. This also applies to encryption but that’s a much more context dependent decision.

[ Parent | on HN ]

Re:A little SSH file copy magic at the command line by m0zg on Friday October 11, 2019 @07:05AM (#21223192)

Just about any machine can gzip at > 120MB/s, just use lower compression settings. You can compress even faster with e.g. zstd, as well.

[ Parent | on HN ]

Re:A little SSH file copy magic at the command line by wang_li on Thursday October 17, 2019 @03:42PM (#21285653)

That doesn't seem to be the case. A relatively fast machine can't gzip anywhere near 120 MB/s unless the data ultra compressible. Zstd is multithreaded, but single threaded it's speed isn't that fast either.

    $ lscpu | grep name
    Model name:            Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz

    $ time dd if=/dev/zero of=/dev/null bs=10240 count=102400
    102400+0 records in
    102400+0 records out
    1048576000 bytes (1.0 GB) copied, 0.145168 s, 7.2 GB/s

    real    0m0.147s
    user    0m0.036s
    sys     0m0.111s

    $ time dd if=/dev/zero bs=10240 count=102400  | gzip > /dev/null
    102400+0 records in
    102400+0 records out
    1048576000 bytes (1.0 GB) copied, 5.36061 s, 196 MB/s

    real    0m5.364s
    user    0m5.317s
    sys     0m0.472s

    $ time dd if=/dev/urandom bs=10240 count=102400 | gzip > /dev/null
    102400+0 records in
    102400+0 records out
    1048576000 bytes (1.0 GB) copied, 27.3478 s, 38.3 MB/s

    real    0m27.352s
    user    0m27.169s
    sys     0m10.119s

    $ time dd if=biggestwords bs=10240 count=102400  | gzip > /dev/null
    10642+1 records in
    10642+1 records out
    108981378 bytes (109 MB) copied, 6.66068 s, 16.4 MB/s

    real    0m6.669s
    user    0m6.667s
    sys     0m0.249s

    $ time dd if=biggestwords  bs=10240 count=102400 | gzip -1 > /dev/null
    10642+1 records in
    10642+1 records out
    108981378 bytes (109 MB) copied, 1.53539 s, 71.0 MB/s

    real    0m1.540s
    user    0m1.522s
    sys     0m0.138s


    $ time dd if=biggestwords of=/dev/null bs=10240 count=102400
    10642+1 records in
    10642+1 records out
    108981378 bytes (109 MB) copied, 0.0407698 s, 2.7 GB/s

    real    0m0.043s
    user    0m0.007s
    sys     0m0.036s

Biggestwords is simply /usr/share/dict/words concatenated to itself 22 times so as to provide ~100 MB of data.

If you can get good enough compression it might still be worthwhile to compress. But it's pretty rare in my experience.

[ Parent | on HN ]