Given that distributed transactions necessitate distributed coordination, it would seem that there is a fundamental tradeoff between scalable performance and support for distributed transactions. Indeed, many practitioners assume that this is the case. When they set out to build a scalable system, they immediately assume that they will not be able to support distributed atomic transactions without severe performance degradation.
This is in fact completely false. It is very much possible for a scalable system to support performant distributed atomic transactions.
In a recent paper [http://cs-www.cs.yale.edu/homes/dna/papers/fit.pdf], we published a new representation of the tradeoffs involved in supporting atomic transactions in scalable systems.
Do you use 2PC or something like Raft to handle distributed transactions?
From the docs it sounds like everytime a node joins/leaves the cluster goes into a (brief?) unstable mode and failures can happen, that doesn't sound great.
We indeed stopped using LevelDB. It has been used for a while but we couldn't work around some speed issues. LevelDB is good for many scenarii when properly configured, though.
The database is multilayered, LevelDB was the last layer, most parallel operation were managed by a massively parallel in-memory database which used LevelDB as a backend.
We do 2PC to handle distributed transactions. Raft breaks our consistency model.
When a node joins/leaves the cluster you might have some transient failures which are generally absorbed by the protocol.
I hope I answered your questions and feel free to give it a spin next week, we're about to release a major update!
And before us FoundationDB delivered the same feature (although implemented a bit differently).
The truth is that there is a market for databases with "unreliable" writes and there are easier to design. That's why you see a lot of them.