News for hackers. Stuff that matters.
Re:Downloading files from S3 with multithreading and Boto3 by aloer on Saturday April 10, 2021 @08:03PM (#26766474)
Extreme case can be when you make heavy use of s3 for ec2 workloads and want to saturate your instance connection. This can go up to 40g afaik

Edit: I read this recently and if I remember correctly there’s a limit of like a thousand parallel connections to s3

AWS internally probably has higher limits for some of their services, e.g. when you query data in s3 with Athena

Re:Downloading files from S3 with multithreading and Boto3 by __turbobrew__ on Saturday April 10, 2021 @08:08PM (#26766499)
Make sense, thank you. Wouldn't you want to multiprocess the downloads instead of multithread? I imagine you would run into the Python global interpreter lock before being able to push through 40gb?
Re:Downloading files from S3 with multithreading and Boto3 by ak217 on Saturday April 10, 2021 @08:54PM (#26766681)
Yes, you would. The CPU and memory overhead of multiprocessing for this application is why we ended up migrating away from boto3 and to the AWS Go SDK for this specific purpose (https://github.com/chanzuckerberg/s3parcp as I mentioned in another comment). We still use boto3 in other areas, but for maxing out the network connection, golang is far more scalable.
Re:Downloading files from S3 with multithreading and Boto3 by Galanwe on Saturday April 10, 2021 @09:57PM (#26766926)
Using multiprocessing I've been able to quite easily saturate a 20GBps Ec2 NICs in python. https://github.com/NewbiZ/s3pd

There is no reason why multiprocessing for IO in python would use _crazily_ more memory than in an other language, when done properly.