“For instance, on a cluster of five m3.large nodes, an even mixture of processes performing single-row inserts and selects over a few hundred rows pushed ~40 inserts and ~20 reads per second (steady-state).”

If someone can explains those abysmal low numbers ...

Well, those numbers, if I remember correctly, refer to insert rate in the face of some extreme amounts of contention - tons of overlapping reads and writes (that's how the test was trying to trigger consistency violations). And moreover, they were measured while the Jepsen test framework was messing with the cluster. Contention is a problem for every database, and particularly so for CRDB. In the absence of contention, we routinely see thousands of queries per second per node.

Since the time of that analyses, we have done a significant amount of work for speeding up these high-contention scenarios, with quite dramatic differences in some cases we looked at. We pretty much changed our transaction execution model from a more "optimistic" one where transactions can abort each other and induce thrashing, to something resembling more the traditional row locks. So, hopefully, even for these atypical uses cases we should generally perform much better.

"These, of course, are pitiful numbers for a database. However, it’s important to note that these numbers are not representative of what real-world applications can expect to see from CockroachDB.

First, this is a test in which the Jepsen nemesis is continuously tweaking the network configuration to cause (and heal) partitions between different nodes. You don’t commonly see behavior like this in a real world deployment.

Second, tests like these deliberately try to reach high levels of contention, in which many transactions are trying to operate on the same keys. In real-world situations contention tends to be much lower as each user is operating mainly on their own data. The performance of all transactional systems suffer under high contention (although CockroachDB’s optimistic concurrency model fares worse than most)"

Reposting my comment from when this came up on our 1.0 launch:

IIRC, in the case that aphyr refers to for these specific numbers, the reads are scans that span multiple shards[1], while the writes are writes to single shards.

[1] even though aphyr says it's just a hundred rows, the tables are split into multiple shards because aphyr in this case was specifically testing our correctness in multi-shard contention scenarios. In production you wouldn't have multi-shard reads crop up until you were doing scans for tables that were hundreds of thousands of rows in size[2]. It's easier to picture if you think of this performance speed difference in the scenario where you are doing full table scans spanning multiple shards on multiple computers while the underlying rows are being rapidly mutated by contending write transactions. The transactions get constantly aborted to avoid giving a non-serializable result, and performance is suffering. We agree that the numbers in this contention scenario are too low, and we are actively working on high-contention performance (and performance in general) leading up to our 1.0 release[3].

[2] Specifically, we break up into multiple shards when a single shard exceeds 64mb in size.

[3] You can follow along on one of the PRs that address this specific performance issue here: https://github.com/cockroachdb/cockroach/pull/13501

CockroachDB peeps,

you guys should really publish a new blog post with updated performance benchmarks.

Super low throughput numbers is deterring a lot of potential users, imho.

Hehe ;) Not to mention—it's Postgres compatible!!

RethinkDB also passes jepsen.

For single-key operations only, but that is still much better than the average system he's looked at.

It's also defunct so... :)

Open Sourced is definitely different than defunct...

Without a company sponsoring and providing support, it is effectively dead for any serious commercial use.

I know I know I was just kidding :D

I am a little hesitant using a database that is younger than 10 years of age (same goes to programming languages, frameworks, etc.).