News for hackers. Stuff that matters.
Re:Pipe Viewer by JayGuerette on Wednesday October 19, 2022 @10:46AM (#33263004)
pv is a great tool. One of it's lesser known features is throttling; transfer a file without dominating your bandwidth:

pv -L 200K < bigfile.iso | ssh somehost 'cat > bigfile.iso'

Complete with a progress bar, speed, and ETA.

Re:Pipe Viewer by smcl on Wednesday October 19, 2022 @10:50AM (#33263054)
Oh damn that's neat I never thought to use `ssh` directly when transferring a file, I always used `scp bigfile.iso [email protected]:path/in/destination`
Re:Pipe Viewer by fbergen on Wednesday October 19, 2022 @11:03AM (#33263264)
Also see `scp -l 200 bigfile.iso [email protected]:path/in/destination`

from man page:

-l limit

Limits the used bandwidth, specified in Kbit/s.

Re:Pipe Viewer by MayeulC on Wednesday October 19, 2022 @01:42PM (#33265485)
Also you probably shouldn't use scp. rsync and sftp have mostly the same semantics.

    rsync --bwlimit=2OOK bigfile.iso [email protected]:path/in/destination
    sftp -l 200 bigfile.iso [email protected]:path/in/destination
Although it seems that scp is becoming a wrapper around sftp these days:

https://www.redhat.com/en/blog/openssh-scp-deprecation-rhel-...

https://news.ycombinator.com/item?id=25005567

Re:Pipe Viewer by tyingq on Wednesday October 19, 2022 @01:41PM (#33265471)
A similar trick that's nice is piping tar through ssh. Handy if you don't have rsync or something better around.

tar -cf - some/dir | ssh remote 'cd /place/to/go;tar -xvf -'

Re:Pipe Viewer by Twirrim on Wednesday October 19, 2022 @04:56PM (#33267820)
I love this trick. I was dealing with some old solaris boxes something like 15 years ago when I learned you could do this. I couldn't rsync, and had started off SCP'ing hundreds of thousands of files across but it was going to take an insane length of time. Asked one of the other sysadmins if they knew a better way and they pointed out you can pipe stuff in to ssh for the other side too. Every now and then this technique proves useful in unexpected ways :)
Re:Pipe Viewer by dspillett on Wednesday October 19, 2022 @06:37PM (#33268738)
Similarly, though useful less often these days, using -B/--buffer-size to increase the amount that it can buffer. If reading data from traditional hard drives, piping that data through some process, and writing the result back to the same drives, this option can increase throughput significantly by reducing head movements. It can help on other storage systems too, but usually not so much so.