Re:We migrated from AWS to GCP with minimal downtime
by levkk on Friday June 07, 2024 @02:00PM
(#40611159)
That remains a mystery for now. We measured upload speeds to S3 to be upwards of 700MB/second (still not 12,800MB/second you'd hope for, but S3 has to write to disk(s)). Download speeds from S3 were low - very low in fact, somewhere around 3-4MB/second when multiple files are downloaded concurrently, and a maximum of again ~30MB/second.
I think you're right and high latency (~40ms) between endpoints triggered some bad TCP behavior. Next time we do something like this, I'll definitely look up how to tune those settings.
Re:We migrated from AWS to GCP with minimal downtime
by klabb3 on Friday June 07, 2024 @02:21PM
(#40611409)
> I think you're right and high latency (~40ms) between endpoints triggered some bad TCP behavior.
40 ms, 200MB/s. The BDP limit for those numbers is 7.6 MB (if I’m holding it right) which is very close to tcp_wmem/tcp_rmem max on Debian like distros, so that sounds about right. Linux can be quite stingy with buffers by default. Easy to increase though!
Since both endpoints are controlled by you, you should be able to tune the tcp buffers. In either case RTT and iperf3 with dozens concurrent tcp conns would be the first step to determine a baseline for what could be expected.
Author if you are here that data would be very interesting to know.