S3, as you know, is a key-based object store, with a hack that allows directories ('/' is just part of the key name, and is filtered) So its less of a deficiency, more of a tradeoff they went with to get performance/uptime. Enumerating the keys on an S3 bucket is pretty slow, so it makes any kind of listing operation for a follow on system slow.

Having a metadata cache is sensible option, especially if you have complete control of a bucket and are able to be canonical. (ie everything talks to your DB to get keynames)

But! what you can't do write to the middle of a file, you need to upload the whole thing again. This is not a problem for a lot of workflows, but for POSIX filesystems thats going to cause problems. You can seek to the middle of a file, write to it, the client will upload it to S3. What happens if another system modifies that file when you are uploading? Sounds like a locking nightmare.

In the case of JuiceFS the file is split into chunks and those are the objects uploaded to S3 - treating it like a high latency block device.

The downside is that metadata here isn't just a cache, it is necessary to operate in this fashion and must be backed up.