News for hackers. Stuff that matters.

throwaway340953

Re:Downloading files from S3 with multithreading and Boto3 on Saturday April 10, 2021 @08:34PM (#26766607)
Depends where you're downloading from. S3 single stream GET throughput is throttled to ~40MB/sec. An HDD can write at ~200MB/sec and SSDs can write ~500MB/sec. NICs on old EC2 instances can do 10Gbps and the new ones can do 25Gbps.

If you're on a home internet/mobile connection downloading large files, a single download will likely saturate your connection. If you're on an EC2 instance, you should be able to do 10-100x better parallelizing.

Source: I used to work on S3.