Re:S3 is files, but not a filesystem
by tsimionescu on Sunday March 10, 2024 @02:02PM
(#39661122)
That's 5s of a 15m duration. I don't think it matters in the least.
Re:S3 is files, but not a filesystem
by anonymous-panda on Sunday March 10, 2024 @03:46PM
(#39662017)
Depends how you’re iterating. If your iterating by hierarchy level, then you could easily see this being several orders of magnitude more requests.
I don’t think btrees would help unless you’re doing directory traversals, but even then I suspect that’s not that beneficial as your bottleneck is going to be the network operations and exposed operations. Ultimately, file listing isn’t that critical a use case and typically most use cases are accomplished through things like object lifecycles where you tell S3 what you want done and it does it efficiently at the FS layer for you.