How to transfer large amounts of data via network

Re:How to transfer large amounts of data via network by batbomb on Thursday February 05, 2015 @04:34PM (#9006279)

For bulk transfer of many files, and especially transfer over a local/nearby network that may be hold true, but as a general practice, and especially for serial/one off transfer of very large files, GNU parallel won't help. However, all the tools mentioned multiplex connections. They are better for transferring individual files, and you can also use them in parallel.

A combination of your network, a CPU threads, frame size, etc... are your ultimate bottleneck when you are transferring very large files.

We've transferred exabytes using bbcp and GridFTP. bbcp is very easy to use once it's set up.

At some point, you run into much different issues when trying to routinely transfer very large files across continents at speeds greater than 10Gbps.

[ Parent | on HN ]

Re:How to transfer large amounts of data via network by anon4 on Thursday February 05, 2015 @05:40PM (#9006630)

For bulk transfer, the absolute fastest I've seen is piping tar through netcat and doing the reverse on the receiving end - on a 10-gigabit lan that results in transfer at the hdd speed. That was between my personal machines with consumer-grade SATA hard disks. The situation probably changes once you add hops and have multiple disks to read from at once.

[ Parent | on HN ]

Re:How to transfer large amounts of data via network by rsync on Thursday February 05, 2015 @11:35PM (#9007749)

bbcp is a supported protocol at rsync.net.

Just saying.

[ Parent | on HN ]