News for hackers. Stuff that matters.
Re:Downloading files from S3 with multithreading and Boto3 by twotwotwo on Saturday April 10, 2021 @08:20PM (#26766552)
When you have a lot of small files, the latency on each operation, though not huge in absolute terms, can end up taking up more time than the data transfer and whatever other work is being done on the content.

(At work, we had an upload job with ~800k files, ranging from <1kb to >100kb. I looked at rearranging how we stored things to avoid small files, but it ended up a straighter shot to continue to use little files but use a worker pool to make the transfer parallel.)

Re:Downloading files from S3 with multithreading and Boto3 by twotwotwo on Monday April 12, 2021 @05:02PM (#26785151)
(Er, some files >100MB not >100KB. If they maxed out around 100KB we probably wouldn't have picked S3 to store them!)