News for hackers. Stuff that matters.
Re:PolarFS: Alibaba Distributed File System for Shared Stora... by KaiserPro on Wednesday August 22, 2018 @06:51AM (#17817743)
The hardest part of a Distributed File system (and I mean File system here) is managing the Meatadata (where a file is, where the directory is, who last did something to it.)

Lustre, GFS2 and GPFS all have centralised metadatstores, which is both a boon and a drawback.

What I can't figure out is what they've done here. It appears like metadata is stored in a special partition ("journal") which is shared? But there is a control process as well.

Re:PolarFS: Alibaba Distributed File System for Shared Stora... by jamesblonde on Wednesday August 22, 2018 @07:26AM (#17817885)
It doesn't look like PolarFS has distributed metadata.

HopsFS (HDFS derivative work) stores distributed metadata, enabling 16X throughput improvements over HDFS - https://www.usenix.org/conference/fast17/technical-sessions/... http://www.hops.io Technically, the hard part is correct concurrent and consistent operations across shards (partitions).

Disclaimer: one of the designers of HopsFS.

Re:PolarFS: Alibaba Distributed File System for Shared Stora... by ddorian43 on Wednesday August 22, 2018 @08:14AM (#17818113)
What do you think are some gotchas/limitations when using mysql-ndb-cluster ?
Re:PolarFS: Alibaba Distributed File System for Shared Stora... by jamesblonde on Wednesday August 22, 2018 @08:20AM (#17818136)
It's become very stable with the years. It's somehow never penetrated the silicon valley echo chamber, probably due to instability in its earlier years. Operationally, it's got most things you need - on-line add node, rolling upgrades, monitoring, etc.

We use NDB to store both the metadata in-memory, but also to store small files on NVMe disks. We had talks with lots of other DB vendors, but, frankly, none of them have high performance support for cross-partition transactions, which is needed. DBs like VoltDB, MemSQL, NuoDB all have promise, but serialize cross-shard operations.

Re:PolarFS: Alibaba Distributed File System for Shared Stora... by ryanworl on Wednesday August 22, 2018 @08:31AM (#17818196)
Have you explored using FoundationDB? It doesn't have a SQL interface (anymore), which might mean it isn't suitable if SQL is required.
Re:PolarFS: Alibaba Distributed File System for Shared Stora... by jamesblonde on Wednesday August 22, 2018 @08:41AM (#17818251)
Not yet, it's only recently been open-sourced. We have a startup now commercializing Hops, called Logical Clocks (of course), so we're busy with that.
Re:PolarFS: Alibaba Distributed File System for Shared Stora... by ryanworl on Wednesday August 22, 2018 @08:54AM (#17818337)
The community is very helpful over on the forums (https://forums.foundationdb.org), so if you do end up having time to check it out, come on over! I'm sure the team at Apple (I'm not affiliated with them) would love to talk about it.
Re:PolarFS: Alibaba Distributed File System for Shared Stora... by ddorian43 on Thursday August 23, 2018 @02:35AM (#17825071)
They're against coprocessors (having functions in the db) so you'll always be somewhat limited. Doing with shared-memory-etc will be hard.

Also you can shard many tables as hash, so most hot-path transactions be inside a server, which you can't guarantee with range-sharding.

Re:PolarFS: Alibaba Distributed File System for Shared Stora... by gnufx on Wednesday August 22, 2018 @09:18AM (#17818455)
With respect to metadata, the premise of PVFS^WOrangeFS is that providing full POSIX semantics (specifically locking?) makes the issues even worse, so it doesn't try, to decent effect. That may rule it out, for some applications like databases, though.

Lustre has DNE for distributed metadata now. Presumably multi-tenancy would be important in this sort of application.