I don’t think btrees would help unless you’re doing directory traversals, but even then I suspect that’s not that beneficial as your bottleneck is going to be the network operations and exposed operations. Ultimately, file listing isn’t that critical a use case and typically most use cases are accomplished through things like object lifecycles where you tell S3 what you want done and it does it efficiently at the FS layer for you.
Indeed there’s every reason to believe that a modern file system would perform significantly faster if the hierarchy was implemented as a prefix filter than actually maintaining the hierarchical data structures (at least for most operations). You can guess that this might be the case that file creation is extremely slow on modern file systems (on the order of hundreds or maybe thousands per second on a modern NVME disk that can otherwise do millions of IOPs and listing the contents of an extremely large directory is exceedingly slow)
I think the previous poster could/should have said, "It is not a hierarchical file system and has no concept of directories." where I added the word "hierarchical".
But it's also pretty obvious that was the point.
For example, you could implement POSIX directory semantics on top of S3. About the only POSIX filesystem API you couldn’t implement it append / overwrite (well you could but it might be prohibitively expensive).
But that illusion breaks when you try to do operations you usually do with/on directories.
After reading that, I now understand your comment.
Deleting an S3 bucket is nontrivial!
You can't delete a bucket with objects in it. And you can't just tell S3 to delete all the objects. You need to send individual API requests to S3 to delete each object. Which means sending requests to S3 to list out the objects, 1000 at a time. Which takes time. And those list calls cost money to execute.
This is a good summary of the situation: https://cloudcasts.io/article/deleting-an-s3-bucket-costs-mo...
The fastest way to quickly dispose of an S3 bucket turns out to be to delete the AWS account it belongs to.
The article has a mistake about this too: expirations do NOT count as lifecycle transitions and you don't get charged as such. You will, of course, get charged if you prematurely delete objects that are in a storage class with a minimum storage duration that they haven't reached yet. This is what they're actually talking about when they mention Infrequent Access and other lower tiers.
Stop being silly, as if 1 vs 2 API calls matters. You should empty large buckets with lifecycle policies. It's trivial.
Imagine you are using windows for the first time and you want to delete a directory, so you find an answer on Serverfault that explains that to do so you need to spin up a COM object that marks the directory for deletion, then the next day comes back and deletes it.
You might be inclined to say ‘that seems overly complicated’.
The original answerer is confused though. ‘It’s trivial, stop being silly. Can you think of a simpler way to delete a directory?’
Do you see now why I thought the ‘non triviality’ of deleting an S3 bucket was perhaps relevant in a discussion on an article about why S3 is both simpler and more complex than a file system?
And why your approach might not actually be making the case for it being as simple as you think?
This is only needed if you have a huge (100 million+) bucket, at which point you should be experienced with s3, otherwise you can just click the big, clear and obvious “empty bucket” button on the console.
It's common to store metadata on DynamoDB where it can be queried, and just have whatever arbitrary links to the values in the buckets.
There's no "recursive" nature to S3 buckets. "Listing a directory" is simply listing keys by a prefix.
So list by the upper-most prefix that you want. If you have 1,000,000 files, it will take 1,000 API calls to list everything.
If each call takes 1s (I have no idea what your latency to the S3 bucket region is), then it will indeed take 15 min.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...
They fixed it already, it's called DynamoDB. With some SQS and Lambda glue you can index your S3 content in any way you want for later retrieval.
I was taken aback by this recently. At my coworkers request, I was putting some work into a script we have to manage assets in S3. It has a cache for the file listing, and my coworker who wrote it sent me his pre-populated cache. My initial thought was “this can’t really be necessary” and started poking.
We have ~100,000 root level directories for our individual assets. Each of those have five or six directories with a handful of files. Probably less than a million files total, maybe 3 levels deep at its deepest.
Recursively listing these files takes literally fifteen minutes. I poked and prodded suggestions from stack overflow and ChatGPT at potential ways to speed up the process and got nothing notable. That’s absurdly slow. Why on earth is it so slow?
Why is this something Amazon has not fixed? From the outside really seems like they could slap some B-trees on the individual buckets and call it a day.
If it is a difficult problem, I’m sure it would be for fascinating reasons I’d love to hear about.