News for hackers. Stuff that matters.
Re:Downloading files from S3 with multithreading and Boto3 by throwaway340953 on Saturday April 10, 2021 @08:34PM (#26766607)
Depends where you're downloading from. S3 single stream GET throughput is throttled to ~40MB/sec. An HDD can write at ~200MB/sec and SSDs can write ~500MB/sec. NICs on old EC2 instances can do 10Gbps and the new ones can do 25Gbps.

If you're on a home internet/mobile connection downloading large files, a single download will likely saturate your connection. If you're on an EC2 instance, you should be able to do 10-100x better parallelizing.

Source: I used to work on S3.

Re:Downloading files from S3 with multithreading and Boto3 by haimez on Saturday April 10, 2021 @09:27PM (#26766818)
FWIW: that 40MB single stream limit doesn’t seem to exist from my empirical measurements in the last two years. When did you work on S3?
Re:Downloading files from S3 with multithreading and Boto3 by Galanwe on Saturday April 10, 2021 @09:51PM (#26766904)
It does exist, but it's 80MB/s nowadays.
Re:Downloading files from S3 with multithreading and Boto3 by Galanwe on Saturday April 10, 2021 @09:50PM (#26766898)
> S3 single stream GET throughput is throttled to ~40MB/sec

More like 80MB/s