throwaway340953
throwaway340953 on Hacker News
Re:Downloading files from S3 with multithreading and Boto3
on Saturday April 10, 2021 @08:34PM (#26766607)
Depends where you're downloading from. S3 single stream GET throughput is throttled to ~40MB/sec. An HDD can write at ~200MB/sec and SSDs can write ~500MB/sec. NICs on old EC2 instances can do 10Gbps and the new ones can do 25Gbps.
If you're on a home internet/mobile connection downloading large files, a single download will likely saturate your connection. If you're on an EC2 instance, you should be able to do 10-100x better parallelizing.
Source: I used to work on S3.