As you say there is no free lunch with distributed filesystems. Application programmers have to program their way around the fact that something like posix atomic writes with multiple writers is never going to work, and that the only way to get reasonable efficiency out of the thing is to defer work until the file is closed.
Per CAP, it's addressed by different meta engines (CP system, Redis, MySQL, TiKV) and also different object stores (AP system). When the meta engine is not available, the operation to JuiceFS will be blocked for a while and finally it returns EIO. When object store returns 404 (object not found), which means it's not consistent with the meta engine, it will be retried for a while, may return EIO if it's not recovered.
The file format is carefully designed to workaround the consistency issue from object store and local cache. Any part of data is written into object store and local cache with unique ID, so you will not go stale data once the metadata is correct [1].
Within a mount point, JuiceFS provides read-after-write consistency. Across clusters, JuiceFS provides open-after-close consistency, which should be enough for most of the applications, also provide good balance between consistency and performance.
[1] https://juicefs.com/docs/community/architecture/#how-juicefs...
It's incredibly hard to make a distributed posix compatible filesystem since you run into CAP. I believe (but am not certain) you are caching locally in violation of Posix or you are signing up for arbitrarily long stalls and a ton of latency on every read/write. (I'm not certain because I'm not sure what Posix specifies wrt stale reads and other cache consistency requirements between sync's)
It would be interesting to hear what the tradeoffs are here, but assuming they are explicit and can be designed around this seems very useful.