How we built a new, fast file transfer protocol

Re:How we built a new, fast file transfer protocol by mcharawi on Saturday March 26, 2022 @02:46PM (#30814230)

Thanks for taking the time to read it! To address your concerns:

1. To give you an idea of the speed improvements, we transferred a 2GB file between Ohio and Singapore on AWS and were able to transfer it in 0:26 (seconds) using our protocol, vs 2:15 for SCP.

2. The basis for improvement is taking into account the changes in round-trip-time for a particular network path; these temporary increases are used as the primary congestion signal.

We are not using error correcting codes, which are good for preventing the retransmission of packets but do not address the underlying problem of avoiding congestion in a network.

[ Parent | on HN ]

Re:How we built a new, fast file transfer protocol by koprulusector on Saturday March 26, 2022 @06:09PM (#30815810)

Can I ask a dumb question? Why SCP and not rsync?

[ Parent | on HN ]

Re:How we built a new, fast file transfer protocol by metadat on Saturday March 26, 2022 @07:14PM (#30816198)

There is no meaningful difference, at the end of the day they'll both be fine for measuring the base case of single-threaded TCP connection performance.

I prefer rsync as well, but SCP works on more machines by default (no dependency on rsync being installed on the target host).

[ Parent | on HN ]

Re:How we built a new, fast file transfer protocol by Straw on Saturday March 26, 2022 @04:28PM (#30815081)

How much was SCP affected by TCP buffer size tuning?

[ Parent | on HN ]