I did a performance evaluation for a project recently and CockroachDB while awesome to setup and operate, was painfully slow. Couldn't get more than a couple thousand writes per second on a cluster consisting of more than half a dozen nodes all in the same datacenter. There seemed to be a huge amount of write amplification (nearly two orders of magnitude) and the load distribution across nodes wasn't really even either.

Sadly had to give up on CockroachDB. They've got good engineers on their team and have some nice funding as well so I'm hopeful that they'll make big improvements in this department eventually. But it'll take quite a lot of work to bring it up to speed - excuse the pun :)

Oh and I don't think Go is any limiting factor here. It's more of an architectural/engineering issue.

I would argue that data structure and system architecture play much more of a role than language choice. Scylla's claims only make sense for datasets that are completely in-memory. Cassandra explicitly targets much larger than memory datasets, and so your bottleneck should be disk throughput not CPU. If you plan to keep everything in memory and just snapshot to disk, you should probably change your choice of data structures.

We're also talking about wide-area storage. Any latency from Go over C++ should be negligible: the Go GC is sub-ms latency now and will just get better [1] while for inter-DC network latency you can expect 50-2/300 ms latencies. The GC just doesn't matter here. Go also gives you control over memory layout, and has an ever-improving compiler [2]. C++ would only really be necessary for sub-ms tail latency systems which are really rare.

I think the gap between BigTable and Hbase is likely due to Google spending more time optimizing BigTable. They do use it for basically everything not built on Spanner.

[1] See recent activity on https://github.com/golang/go/issues/10958 [2] Go has a ways to match GCC, but you can work around the language for now to get close to C. If you want to improve things here, maybe contribute to gccgo or gollvm so Go users can get all of LLVM & GCC's optimizations.

Note that the underlying data store where a lot of the byte churning happens is RocksDB, written in C++. Also the Go/C++ interface in CockroachDB is coarse to reduce overheads.

In that I would be curious on RocksDB vs LMDB. There was an issue about this but it didn't progress: https://github.com/cockroachdb/cockroach/issues/14#issuecomm...

For some reason(s) nobody is using LMDB on distributed dbs (only know actordb). While for single-nodes there are many cases that have switched and been better.

I know and that you take special care to bulk as much stuff to it for lower overhead. But then how much of % speed is left into GO ? Say GO is 2x slower, and the db is bottlenecked on cpu/ram, you're using 2x servers. While if it's bottlenecked on disk it's rocksdb which you can't make faster by using a different language.

Like even TiDB (mysql protocol, golang) uses TiKB (kv in rust,rocksdb) to make it faster.

Relevant: blog post from the TiDB team about why they didn't choose Go for the core distributed storage engine: https://pingcap.github.io/blog/2017/05/27/rust-in-tikv/#go

Agreed. I love Go, but a database is one of the things in the stack that ought to be as optimized as possible in my opinion. C++ still seems preferable in these types of applications. At the same time, if the underlying architecture is enough of an improvement over older systems, it's still an upgrade.

You can't optimize anything with C++ if your database is stuck waiting for consensus with nodes 100 ms away. Close to the metal and low level languages are really poor choices here, even Go is too low level for CP databases.

Yes, but you'll have lower overhead on each query/request and more concurrency ? For each query the bottleneck is the network which you can't fix, but you can fix the 2,3,etc bottlenecks which are inside 1 machine (cpu/ram/disk).

Yeah...I'm not really aware of a higher level language that mystically makes networking faster. As far as I'm aware, the most efficient network protocols tend to be implemented at a low level, too.

> C++ still seems preferable in these types of applications

Agreed, but until there are some published performance metrics, I think it's an assumption more than a truth. Premature optimization...

Well we already have metrics on what the languages themselves can do. And given that performance will definitely matter in this domain, it's not a premature optimization at all. A database isn't something that can be written easily enough to simply rewrite it in another language later as an optimization.