News for hackers. Stuff that matters.

emasquil

Sorry about that! Next time I work on my blog I'll take that into account and find a better theme/color palette. In the meantime you can change it to light mode (clicking on the top right button) and there you have black text on white background.
I just mentioned the issue because a colleague of mine did some quick tests with aioboto3 and he told me it was downloading the files sequentially. We thought it might be related to that.

You made a great point about going async if you need a working command channel.

Re:Downloading files from S3 with multithreading and Boto3 on Saturday April 10, 2021 @09:49PM (#26766894)
Unfortunately I haven't. If someone wants to benchmark this solution I'd be glad to hear about that!
Re:Downloading files from S3 with multithreading and Boto3 on Saturday April 10, 2021 @09:40PM (#26766865)
In this case I needed to download about 1k files of size ~ 10mb. Doing this with multithreading was so much faster than downloading all the files sequentially.
Re:Downloading files from S3 with multithreading and Boto3 on Saturday April 10, 2021 @09:36PM (#26766847)
In this case I just wanted to keep it simple and do it the "native" way. I've been hearing a lot about aioboto3 and I'll surely check it out. However, it seems to have some limitations like https://github.com/boto/botocore/issues/458