HopsFS (HDFS derivative work) stores distributed metadata, enabling 16X throughput improvements over HDFS - https://www.usenix.org/conference/fast17/technical-sessions/... http://www.hops.io Technically, the hard part is correct concurrent and consistent operations across shards (partitions).
Disclaimer: one of the designers of HopsFS.
We use NDB to store both the metadata in-memory, but also to store small files on NVMe disks. We had talks with lots of other DB vendors, but, frankly, none of them have high performance support for cross-partition transactions, which is needed. DBs like VoltDB, MemSQL, NuoDB all have promise, but serialize cross-shard operations.
Also you can shard many tables as hash, so most hot-path transactions be inside a server, which you can't guarantee with range-sharding.
Lustre has DNE for distributed metadata now. Presumably multi-tenancy would be important in this sort of application.
Lustre, GFS2 and GPFS all have centralised metadatstores, which is both a boon and a drawback.
What I can't figure out is what they've done here. It appears like metadata is stored in a special partition ("journal") which is shared? But there is a control process as well.