hnkimb3558

What's next after optimizing for Snowflake costs?

I just read up on that issue. We will follow up with updated measurements with CRDB 2.0. The big reason for the discrepancies you measured is likely the fact that each sqlbench write transaction is doing six consecutive consensus writes (incl. commit). With CRDB, you need to parallelize your transaction writes using the `RETURNING NOTHING` syntax, which would likely be a significant improvement. This manual tuning is obviously sub-optimal, and will not be required in the next version of CockroachDB (2.1 in October).

Performance is a very tricky thing to measure in a database. CockroachDB's performance is certainly affected by its consistency model. In particular, CockroachDB handles transactions using serializable isolation, and writes using consensus replication. This means that it's difficult to make an apples to apples comparison for performance between a replicated CockroachDB cluster and a single master MySQL server. However, that is the whole point of a well thought out, standardized benchmark like TPC-C. It creates a pseudo-realistic workload, and mandates the use of transactions with a reasonable isolation level or higher (snapshot isolation), carefully calibrated degrees of contention and failures, and then leaves it up to the database vendor to optimize within those parameters.

When Amazon did this for Aurora, they showed that with their custom storage backplane, they were able to absolutely murder normal RDS-based MySQL in terms of TPC-C throughput. Take a look: https://www.slideshare.net/AmazonWebServices/dat202getting-s.... The bottom right quadrant of that slide shows what happens to a standard MySQL server when given 10,000 TPC-C warehouses to chew on (about 800 GB of TPC-C dataset). The max throughput is supposed to be ~128,000 tpmC. RDS MySQL in their tests was able to do 69. That is quite simply, not impressive. Even Aurora, which is a full 136x the tpmC throughput, is still more than 10x less than the max throughput that should be achieved at that number of warehouses.

CockroachDB can scale out, so while it may have more latency when it performs a simple transaction or millions of simple transactions over a small set of data, it can also easily scale to handle 10,000 TPC-C warehouses, with 126,000 tpmC. If you're going to talk about performance, you really need a serious benchmark, or you're just kidding yourself.

Let me put this another way. While Amazon frequently talks about how Aurora can be scaled to 64TB of data, in the context of TPC-C, that's a risible claim. A maximal instance of Aurora, tuned by AWS, can handle max TPC-C throughput at something between 80GB and 800GB of data (i.e. 1,000 to 10,000 warehouses). It's like they're selling you a pickup truck with a bed they say can handle 64 tons, only if you load it that way, it can only travel 2mph, and can't turn. Caveat emptor!

I think the algorithm is whoever comes up with the most egregious and sticky name wins.

Pete chose the name GIMP (we were mulling calling it "XIMP", for X11 Image Manipulation Program, but Pulp Fiction had recently been released, so...).

I (spencer) chose the name CockroachDB. I guess we have questionable tastes in OSS project naming.

[Post author here] I second the sentiment that you should publish. This would be invaluable.

Post author here.

First off, I really appreciated your frank blog post on the RethinkDB post mortem. The distillation of years of experience is incredibly valuable for us, and I'm sure for many others.

I agree that the database market is crowded with solid offerings. However, I believe that differentiating features do still matter and there will be tremendous growth in the database market for the foreseeable future. In your blog post you listed the metrics of goodness which you optimized for, perhaps incorrectly. You indeed had amazing execution on those original metrics. We have paid RethinkDB the compliment of doing our best to emulate the standards set with simplicity and consistency, in particular. You are also correct that the alternate metrics including timely arrival, palpable speed, and a use case are probably better ones to optimize for in an entrepreneurial setting.

We have been optimizing from the start for a still-small use case, but one which is likely to become a top of mind concern for every major enterprise over the next five years: building global, "multihomed" services. This is something Google has pioneered over the past decade, but which remains an elusive challenge for most everyone else. For an interesting read, check out https://static.googleusercontent.com/media/research.google.c... (tl;dr here: http://highscalability.com/blog/2016/2/23/googles-transition...)

You mention AWS/GCloud as existential risks for a cloud DBaaS offering. I would take that a step further and cite them as the biggest risk to all database companies. We must compete with them both by pushing the boundaries of what the database can accomplish, and by aggressively driving an anti-vendor-lockin message: embracing a proprietary cloud DBaaS offering is an unacceptable risk if there are non-proprietary alternatives.

(Post author here) I'm not incredibly familiar with the ins and outs of MongoDB Inc's business model, but I agree with your assessment. They certainly seem to have embraced both of the OSS business models I described in the post as viable alternatives. We are going to start with just one, and there's still a huge amount on our plate.

MongoDB Inc did have its valuation reduced by some institutional investors. Hard to say whether that was premature or what the impetus was behind their decision. MongoDB is an incredibly well-adopted product that has gotten considerably more capable over the years. I would argue they've had good success with this business model, as building a $1.6B business is a huge accomplishment whether you've got an OSS business model or not.

It would be fair to ask whether they've done the balancing act as well as they might have. They've certainly knocked OSS adoption out of the park. On the other hand, I've heard anecdotally that they waited a long time before introducing enterprise features.

Regardless, I view MongoDB Inc. as a big - and still growing - success, and consider much of what they've accomplished to be worthy of emulation.

I'm not really sure which strategy would benchmark best between 64K chunks, 1M chunks, or even 8M chunks. I think this requires some experimentation. Pushing them all through as 64K chunks has a lot of overhead, and you'd reap the full write amplification. Could you tell me a bit more about your use case? You can email me spencer at cockroachlabs.com.

I've been meaning to work on the "CockroachDB Egg Store" (my disgusting name for a blob storage subsystem) as a Free Fridays side project for a while, but have been distracted with all manner of other enticing options. There aren't enough hours in a week...

(employee/founder here)

Anything under 64K is perfectly reasonable to store in a CockroachDB column. Between 64K and maybe 1M is trending towards trouble. Values greater than this are going to go through CockroachDB like a goat through a python.

Why is this the case? For starters, at the level of RocksDB, values greater than 64K are not jammed into SSTables (to avoid constantly rewriting them during compactions of the LSM tree). Instead, individual files are created. Also, CockroachDB has quite a lot of write amplification, which is generally OK for structured relational data, but becomes progressively more terrible for large blobs. Write amplification comes from the Raft log, as well as RocksDB's write-ahead log.

What we really need is an integrated storage system for immutable blobs, something we're taking very seriously. Roughly half of the original team which built Colossus at Google are working on CockroachDB, so there's some knowledge of how to go about building such a system.

While we're not sure where it would fall on our roadmap, the idea is that large blob values would be efficiently replicated and maintained through a separate subsystem. The blob column itself would just contain a pointer to the blob. The value in tight integration (a single CockroachDB cluster providing both OLTP SQL database as well as a distributed blob store) would be one deployment & admin console, and transactionally consistent blob column values (e.g. no fighting s3 eventual consistency).

Cockroach enforces this guarantee on a per-key basis as opposed to for the entire node. If a key has been read at time t, it may only be subsequently written at time > t. CockroachDB accomplishes this using a timestamp cache at the leader node for a range. But this doesn't cause writes to block. Instead, the write timestamp is pushed to the most recent read + 1 logical tick.

CockroachDB's time api is based on the Hybrid Logical Clocks paper. We credit it on our design doc (https://github.com/cockroachdb/cockroach/blob/master/docs/de...) and in the source code.

This blog post doesn't mention HLC because the explanation didn't require it. HLC boils down to a mechanism for taking the maximum physical wall time across >= 2 nodes, while still being able to provide monotonically increasing time via incrementing the logical component of the hybrid logical timestamp. In the blog post, this is referred to as "taking the maximum timestamp across requests". A segue into HLC would have served only to further complicate the explanation.

The work done by Sandeep Kulkarni, Murat Demirbas, David Alves, Todd Lipcon, Vijay Garg, and others was instrumental to our early design efforts. I am dismayed to see there's a patent pending, though both our design and open source implementation predate the patent filing.

These are conservative upper bounds and are meant to encompass long tail offsets. You can usually get NTP down to < 10ms offsets, but if you rely on that, you'll likely run into problems which end up violating your database guarantees.