Re:PolarFS: Alibaba Distributed File System for Shared Stora...
by ddorian43 on Wednesday August 22, 2018 @08:14AM
(#17818113)
What do you think are some gotchas/limitations when using mysql-ndb-cluster ?
Re:PolarFS: Alibaba Distributed File System for Shared Stora...
by jamesblonde on Wednesday August 22, 2018 @08:20AM
(#17818136)
It's become very stable with the years. It's somehow never penetrated the silicon valley echo chamber, probably due to instability in its earlier years. Operationally, it's got most things you need - on-line add node, rolling upgrades, monitoring, etc.
We use NDB to store both the metadata in-memory, but also to store small files on NVMe disks. We had talks with lots of other DB vendors, but, frankly, none of them have high performance support for cross-partition transactions, which is needed. DBs like VoltDB, MemSQL, NuoDB all have promise, but serialize cross-shard operations.
Re:PolarFS: Alibaba Distributed File System for Shared Stora...
by ryanworl on Wednesday August 22, 2018 @08:31AM
(#17818196)
Have you explored using FoundationDB? It doesn't have a SQL interface (anymore), which might mean it isn't suitable if SQL is required.
Re:PolarFS: Alibaba Distributed File System for Shared Stora...
by jamesblonde on Wednesday August 22, 2018 @08:41AM
(#17818251)
Not yet, it's only recently been open-sourced. We have a startup now commercializing Hops, called Logical Clocks (of course), so we're busy with that.
Re:PolarFS: Alibaba Distributed File System for Shared Stora...
by ryanworl on Wednesday August 22, 2018 @08:54AM
(#17818337)
The community is very helpful over on the forums (https://forums.foundationdb.org), so if you do end up having time to check it out, come on over! I'm sure the team at Apple (I'm not affiliated with them) would love to talk about it.
Re:PolarFS: Alibaba Distributed File System for Shared Stora...
by ddorian43 on Thursday August 23, 2018 @02:35AM
(#17825071)
They're against coprocessors (having functions in the db) so you'll always be somewhat limited. Doing with shared-memory-etc will be hard.
Also you can shard many tables as hash, so most hot-path transactions be inside a server, which you can't guarantee with range-sharding.
HopsFS (HDFS derivative work) stores distributed metadata, enabling 16X throughput improvements over HDFS - https://www.usenix.org/conference/fast17/technical-sessions/... http://www.hops.io Technically, the hard part is correct concurrent and consistent operations across shards (partitions).
Disclaimer: one of the designers of HopsFS.