News for hackers. Stuff that matters.
Re:S3 is files, but not a filesystem by electroly on Sunday March 10, 2024 @02:52PM (#39661534)
The way that you said "recursively" and spent a lot of time describing "directories" and "levels" worries me. The fastest way to list objects in S3 wouldn't involve recursion at all; you just list all objects under a prefix. If you're using the path delimiter to pretend that S3 keys are a folder structure (they're not) and go "folder by folder", it's going to be way slower. When calling ListObjectsV2, make sure you are NOT passing "delimiter". The "directories" and "levels" have no impact on performance when you're not using the delimiter functionality. Split the one list operation into multiple parallel lists on separate prefixes to attain any total time goal you'd like.
Re:S3 is files, but not a filesystem by blakesley on Sunday March 10, 2024 @03:59PM (#39662109)
All these comments saying merely "S3 has no concept of directories" without an explanation (or at least a link to an explanation) are pretty unhelpful, IMO. I dismissed your comment, but then I came upon this later one explaining why: https://news.ycombinator.com/item?id=39660445

After reading that, I now understand your comment.

Re:S3 is files, but not a filesystem by electroly on Sunday March 10, 2024 @05:48PM (#39662782)
I appreciate you sharing that point of view. There's a "curse of knowledge" effect with AWS where its card-carrying proponents (myself included) lose perspective on how complex it actually is.
Re:S3 is files, but not a filesystem by petters on Sunday March 10, 2024 @03:34PM (#39661902)
Yes, this is very good advice and will likely solve their problem