News for hackers. Stuff that matters.
Re:Ask HN: What is your favorite method of sending large files? by ashton314 on Wednesday September 02, 2020 @04:14PM (#24357573)
Netcat:

    $ nc -l 4242 > dest
And then on the sending end:

    $ nc hostname 4242 < file_to_send
This works great when you just need to get a file of any size from one machine to another and you’re both on the same network. Are used this a lot at one of my offices to schlep the files around between a few of the machines we had.
Re:Ask HN: What is your favorite method of sending large files? by unilynx on Wednesday September 02, 2020 @04:38PM (#24357855)
This is especially fun when combined with piping through tar, and adding pv in the mix for a transfer speed "progress" bar.

Fastest way to transfer a collection of files on a local network and doesn't require temporary storage for the archive

Re:Ask HN: What is your favorite method of sending large files? by enriquto on Wednesday September 02, 2020 @05:37PM (#24358452)
piping through tar is my "power move" when I want to steer a new recruit. I just do it casually in front of them... depending on whether they are awed or horrified by that, they'll get more hacky or more formal work.
Re:Ask HN: What is your favorite method of sending large files? by parliament32 on Wednesday September 02, 2020 @05:43PM (#24358517)
You can also pipe through gpg if you want encryption along the way!
Re:Ask HN: What is your favorite method of sending large files? by ahsima1 on Wednesday September 02, 2020 @05:51PM (#24358590)
And through gzip/zstd/etc if you want compression!
Re:Ask HN: What is your favorite method of sending large files? by 3np on Thursday September 03, 2020 @02:19AM (#24361238)
And through tac if you want it in reverse!

(...And my axe!)

Re:Ask HN: What is your favorite method of sending large files? by labawi on Friday September 04, 2020 @01:53PM (#24377261)
Note that gpg compresses by default. Which is thy it's so slow. You may wish to use lzop or zstd but also disable gpg's compression.

I usually use ssh (e.g. ssh -L 3333:localhost:3333) and avoid gpg, though I may use lzop depending on link and CPU speed.

Re:Ask HN: What is your favorite method of sending large files? by enriquto on Wednesday September 02, 2020 @05:58PM (#24358656)
that's what the -C option to ssh is for
Re:Ask HN: What is your favorite method of sending large files? by cellularmitosis on Wednesday September 02, 2020 @06:21PM (#24358881)
lzop if you gotta go fast :)
Re:Ask HN: What is your favorite method of sending large files? by namibj on Thursday September 03, 2020 @03:40AM (#24361683)
zstd easily outruns a gigabit link on most somewhat-compressible data.
Re:Ask HN: What is your favorite method of sending large files? by kortex on Wednesday September 02, 2020 @10:26PM (#24360304)
What's an example of the syntax of this? I remember seeing a graybeard do something like this to huck some files to another machine.
Re:Ask HN: What is your favorite method of sending large files? by waterhouse on Thursday September 03, 2020 @09:41AM (#24363929)
Paraphrased from a real example:

  tar cf - files morefiles | ssh user@dest "tar xf -"
If you want to insert, say, compression:

  tar cf - files morefiles | zstd | ssh user@dest "zstd -d | tar xf -"
Re:Ask HN: What is your favorite method of sending large files? by ashton314 on Thursday September 03, 2020 @01:28PM (#24366397)
Nice. This to me is a great testament of the power of the Unix philosophy: small orthogonal tools that operate on streams of text.
Re:Ask HN: What is your favorite method of sending large files? by Matthias247 on Wednesday September 02, 2020 @04:48PM (#24357972)
and if you are not worried about corruption. You might at least want to run a checksum on both ends and compare them.
Re:Ask HN: What is your favorite method of sending large files? by myself248 on Wednesday September 02, 2020 @05:52PM (#24358595)
Nah, unix nerds don't care about data corruption. That's the responsibility of some other layer, so by definition we're not in trouble if the whole thing blows up by our own negligence.

Checkmate, rsync.

Re:Ask HN: What is your favorite method of sending large files? by gpm on Wednesday September 02, 2020 @06:44PM (#24359078)
Netcat is tcp is "reliable" and has checksums for each packet, as long as the length is right you can be reasonably assured that the file transferred correctly.

Not that extra checksums are a bad idea...

Re:Ask HN: What is your favorite method of sending large files? by Matthias247 on Wednesday September 02, 2020 @09:10PM (#24359947)
TCP indeed has checksums! But they are known not to offer ultimate protection. I had to deal with hosts corrupting TCP streams due to bad hardware before. Which was luckily detected by higher level checksums.
Re:Ask HN: What is your favorite method of sending large files? by kortex on Wednesday September 02, 2020 @10:29PM (#24360313)
Is there a streamlike tool (a la sed, gzip, etc) that applies forward error correction? I've been looking for such a thing but can't find one that scratches the itch.
Re:Ask HN: What is your favorite method of sending large files? by rags2riches on Thursday September 03, 2020 @08:26AM (#24363303)
I've done that many times, often in combination with tar and pv as mentioned in another reply. But I prefer to do the listening on the sending side. Feels more like serving than shoving I guess. Also, I remember needing a -q flag on one of the ends for graceful shutdowns.