> I am skeptical that a networked database with multiple nodes can match the performance of a single master database such as MySQL, PostgreSQL, or SQL Server.

It likely can't unless there is some black magic going on. Single node speed will always be faster. But once you get to that point where a single node chokes on the amount of data or query throughput that you have, you don't really have a choice anymore.

My personal plan is to start with Postgres RDS. Grow until RDS doesn't work anymore, then move to ultra beefy bare metal servers in colocation with AWS Direct Connect. If I ever outgrow an 4x24 core server with 3TB of memory on a RAIDed NVMe disk cluster, I might move to Citus.

For the various distributed database companies out there, I believe the one that will win in the marketplace is the one working or partnering to develop specialized hardware and networking, and then optimizing for it.

> then move to ultra beefy bare metal servers in colocation with AWS Direct Connect.

You will still need another beefy server in another datacenter with configured replication and failover, which is one of the problem Cockroach is trying to solve.

Why not bare metal on AWS? Going to DX is going to give you a conservative 10ms on your calls...

That might be a good intermediate option, but they still don't have instances with 4x24 cpu, or >488GB ram.

Did you look at the x1 options? Up to 128 CPUs and 3,904GB of RAM on the x1e.32xlarge.

Those aren't bare metal. And in terms of database scalability, I would jump for bare metal before beefier servers, due to the fact that the disk is local and you don't have to deal with EBS noisy neighbors.

The x1e.32xlarge has 4TB of SSD in addition to dedicated EBS bandwidth.

I haven't tried it, but IIRC that's only network bandwidth to the block store. Better than the alternative for sure, but I would assume disk access is still noisy unless you have a dedicated drive.

The max IOPS for provisioned EBS is less than 10% the max IOPS of dedicated hardware SSDs.

True, but you do have 4TB dedicated SSD

There is some black magic[0] going on with BedrockDB. They are using a single node at Expensify and getting 4 million queries per second.

[0]: https://news.ycombinator.com/item?id=16118776

A distributed db running on a single node isn't distributed, so that is just showing the performance of SQLite on a single node, which is the point of the thread.

Also bedrock has plenty of problems and is highly unrecommended.

> I believe the one that will win in the marketplace is the one working or partnering to develop specialized hardware and networking, and then optimizing for it.

soo... oracle? :b

You are 100% correct about waiting for several years before using a new database product.

However, if you look at the path Cockroach is taking, they're doing all the right things to shave those years down. They already Jepsen themselves, and they've been doing rigorous testing forever now.

Very exciting.

So did RethinkDB. :(

It's still alive as an open source project, but probably less support now. https://github.com/rethinkdb/rethinkdb

RethinkDB still has a 2 year old critical memory leak that probably won't ever be fixed:

https://github.com/rethinkdb/rethinkdb/issues/5865

And still available as a managed service on Compose: https://www.compose.com/databases/rethinkdb

Touché

Performance is a very tricky thing to measure in a database. CockroachDB's performance is certainly affected by its consistency model. In particular, CockroachDB handles transactions using serializable isolation, and writes using consensus replication. This means that it's difficult to make an apples to apples comparison for performance between a replicated CockroachDB cluster and a single master MySQL server. However, that is the whole point of a well thought out, standardized benchmark like TPC-C. It creates a pseudo-realistic workload, and mandates the use of transactions with a reasonable isolation level or higher (snapshot isolation), carefully calibrated degrees of contention and failures, and then leaves it up to the database vendor to optimize within those parameters.

When Amazon did this for Aurora, they showed that with their custom storage backplane, they were able to absolutely murder normal RDS-based MySQL in terms of TPC-C throughput. Take a look: https://www.slideshare.net/AmazonWebServices/dat202getting-s.... The bottom right quadrant of that slide shows what happens to a standard MySQL server when given 10,000 TPC-C warehouses to chew on (about 800 GB of TPC-C dataset). The max throughput is supposed to be ~128,000 tpmC. RDS MySQL in their tests was able to do 69. That is quite simply, not impressive. Even Aurora, which is a full 136x the tpmC throughput, is still more than 10x less than the max throughput that should be achieved at that number of warehouses.

CockroachDB can scale out, so while it may have more latency when it performs a simple transaction or millions of simple transactions over a small set of data, it can also easily scale to handle 10,000 TPC-C warehouses, with 126,000 tpmC. If you're going to talk about performance, you really need a serious benchmark, or you're just kidding yourself.

Let me put this another way. While Amazon frequently talks about how Aurora can be scaled to 64TB of data, in the context of TPC-C, that's a risible claim. A maximal instance of Aurora, tuned by AWS, can handle max TPC-C throughput at something between 80GB and 800GB of data (i.e. 1,000 to 10,000 warehouses). It's like they're selling you a pickup truck with a bed they say can handle 64 tons, only if you load it that way, it can only travel 2mph, and can't turn. Caveat emptor!

When I tested CockroachDB, I did a single-node and multi-node comparison to Galera. I was ready to accept that multi-node is going to have a performance hit, but what surprised me the most was that the single-node performance was roughly 5-10 times worse with CockroachDB.

https://github.com/cockroachdb/cockroach/issues/17777#issuec...

I just read up on that issue. We will follow up with updated measurements with CRDB 2.0. The big reason for the discrepancies you measured is likely the fact that each sqlbench write transaction is doing six consecutive consensus writes (incl. commit). With CRDB, you need to parallelize your transaction writes using the `RETURNING NOTHING` syntax, which would likely be a significant improvement. This manual tuning is obviously sub-optimal, and will not be required in the next version of CockroachDB (2.1 in October).

A distributed database will never be faster than a single-node RDBMS for simple queries because there is added overhead of coordination between nodes, especially if you're repeatedly writing to the same exact rows.

What distributed databases give you is scalability as data grows, and high-availability for safety. Performance comes from that scale and concurrency when your data and queries fit the model.

Not entirely correct. The distributed database can scale computation horizontally across the nodes in cluster. Single node is better only for the relatively small dataset. If you go higher volumes and velocity, Vertical scaling is still more expensive than Horizontal scaling. If your data is large, with network bandwidth of 1Gbps (or perhaps better) distributed database may choose to replicate it efficiently across the nodes and do scatter-gather (or divide and conquer) query parallelization and also maintain HA. Citus DB is one example of distributed database which does exactly that.

That’s basically what I said...the point is that simple queries on the same keys will never be faster.

There's definitely cases where a distributed database will have higher bandwidth than a single-node database. Since each range is replicated across 3 machines, that means if you have a lot more nodes than 3, the overlap of ranges significantly decreases and you start being able to handle more requests than a single node can.

However, the latency will typically be higher since it always has to talk to at least 3 nodes.

If you consider CRDB's DistSQL which lets you run queries on multiple machines, it might be possible to beat a traditional RDBMS on read latency for large queries.

It would be nice if CRDB could have a mode when 1-3 nodes are active that is similar to since master performance with automatic replication. Then if horizontal scale-out is needed a performance hit could be taken.

I like the automatic replication of CRDB but the vast majority of apps can't afford to take a big performance hit and don't need large horizontal scale-out.

You can run CRDB with only one replica per range. You just lose a lot of reliability if a node goes down.

They also only handle data corruption at the cluster level so there's not really any tools to extract data from a partially corrupted DB on a single node.

I totally agree. So true. Lots of time I ask myself what is the point of compare a single node RDBMS to a distributed database.

Even with MySQL, the replication is still asynchronous from binlog? Is it also the case with postgres?

Yes, postgres has sync and async replication with the ability to even choose a quorum (so that 2 of 3 replicas confirm).

Also has logical replication now, although I wouldnt recommend it because it doesn't support DDL changes yet. Overall it's a solid db choice, but there is still some work involved.

Otherwise I'd recommend SQL Server for a fantastic commercial database that runs cross-platform and has great tooling to make everything easier and some advanced features.

In postgres you can choose if the replication is synchronous or asynchronous. For cascade replication (a slave uses as a master for another slave) the replica is always async.

My general rule: always pick the boring major RDBMS - unless you have a specific reason not to.

High availability with automatic failover is a pretty common reason. To do this with a "boring major RDBMS" removes the boring part and requires leaving the beaten path of the core RDBMS product anyways (or paying lots of money). If multi-master w/ automatic scaling and failover were a first class feature of these boring major RDBMSs, you could have the best of all worlds.

Yeah, I've got a few projects that I'm not doing much where scaling would be an issue, but I really want easy multi-master. I want to be able to have a few database servers and be able to take one offline for maintenance (or because of mishap) with minimal fanfare, without it causing downtime, and without it feeling like I'm performing open heart surgery. There's plenty of ways to fuck up a manual switchover, and I don't want to discover more of them again in production.

Technically, I don't really need multi-master, a "standard RDBMS" with automatic failover would probably fit the bill, but I could not for the life of me figure out any standard (and free) way of doing that for mysql/mariadb or postgresql.

Do you need a distributed database for your use case? Or do you like the new tech? Genuine question, not trying to be snarky.

I do not have a particular project in mind, but think it would be good to have an easily containerizable, replicated OLTP database in my toolkit. I do some work with Entity Framework Core providers and am thinking about trying to implement a provider for CRDB.

There is already some work towards this with http://www.npgsql.org/

I haven't gone back to try the full entity framework recently, but if you find any bugs, please let us know. If you want to roll your own, we're here to help too.

CRDB is using the Postgresql wire transport and already works with npgsql + ef core.

Are you sure? I know that the NPGSQL DB driver works with CRDB. Data types, DDL, and supported features are a big part of EF Core providers though. I have not researched it yet but I was pretty sure the NPGSQL EF Core Provider implemetation was just for PG.

CockroachDB uses the same PostgreSQL wire protocol and SQL dialect so it just looks and works like a pg database. There are some minor differences but DDL and data types are already included in the syntax: https://www.cockroachlabs.com/docs/stable/porting-postgres.h...

It's great if you want to contribute though but I'd recommend just working on the PG EF Core provider which already has some work towards CRDB specific features: https://github.com/npgsql/Npgsql.EntityFrameworkCore.Postgre...

Great, thanks for the info. I have worked with @roji before on EF Core upstream issues, NPGSQL EF Core is very good. I will reach out to him!

It's really to sad RethinkDB gone.

While I keep hearing that the RDMS(Postgres, MySQL...) was built on year of experiences I alwawyas think why we cannot come up with something nicer, more friendlier to the old way. Eventually we will build up knowledge on the new thing and have a system that just as good but also as much friendlier.

Example: when looking at how MongoDB handle replication, it's so easy to just add a new node and have it join cluster without seeding some data and set binlog position (like MySQL).

Or looking at RethinkDB query language, it's just so much easier compare with SQL, at least to me.

Nowsday, I cry whenever I see SQL query. I learn to master it, but life's too short to write those kind of thing.

But, this is a thing in life when you are simply wrong. So find a way to learn it or remain a subpar dev.

> I am genuinely interested in using CockroachDB as a primary datastore but feel like I have been burned too many times by hopping on board with a young database.

> According to crunchbase, Cockroach Labs has raised $53.5m (RethinkDB had only raised $12m). Is there evidence that Cockroach Labs is on track to make money and is going to survive?

I'm in a similar situation: new project, CRDB is a good fit on paper, but I'm unable to get traction even for a proof of concept exercise because of the risk associated with a database backed by a startup. This is at a large, conservative, 300k+ employee company, so I guess it's understandable... but regrettable nonetheless.

See you guys in 5 years ;-)

Becoming a paying customer is a way to help them survive.

When it comes to databases, always wait for the Jepsen test!

https://jepsen.io/

CockroachDB underwent a Jepsen audit by Kyle Kingsbury before its 1.0 release. Check out the blog post[0] and Kyle's analysis[1].

We also run Jepsen tests against the database nightly, ensuring that its isolation guarantees do not regress.

[0]: https://www.cockroachlabs.com/blog/cockroachdb-beta-passes-j...

[1]: http://jepsen.io/analyses/cockroachdb-beta-20160829

Which they've been doing for around 2 years at least

https://www.cockroachlabs.com/blog/diy-jepsen-testing-cockro...

As others said, Cockroach's performance is limited by its consistency model. It is not something that can ever get close to a single master MySQL. This might also become an issue for the company and ultimately prevent it from reaching mass adoption and succeeding, at least commercially.