petermattis
Yes.
> Does it multiplex a single query by dispatching smaller requests to a thread-pool?
Yes, though it depends on the query. Trivial queries (i.e. single-row lookups) are executed serially as that is the fastest way to execute them. Complex queries are decomposed along data boundaries and the query parts are executed in parallel next to where the data is located.
The equivalent of FoundationDB is present inside of CockroachDB: a distributed, replicated, transactional, KV layer. This is where a big chunk of CockroachDB value resides. This is where our use of Raft resides. Pebble lies underneath this.
PS PebblesDB was a research project and is dead as far as I know.
RocksDB MultiGet is interesting. Parallelism is achieved by using new IO interfaces (io_uring), not by using threads. That approach seems right to me. See https://github.com/facebook/rocksdb/wiki/MultiGet-Performanc.... My understanding is that io_uring support is still a work in progress. We experimented at one point with using goroutines in Pebble to parallelize lookups, but doing so was strictly worse for performance. Experimenting with io_uring is something we'd like to do.
From the Rebello paper: > However, on restart, since the log entry is in the page cache, LevelDB includes it while creating an SSTable from the log file.
Pebble and RocksDB both inherited this behavior. The nuance here is that the sstable is then synced to disk and no reads are served until the sync is successful. If the machine were to crash before the sstable was synced, upon restart we'd rollback to the durable prefix of the log.
We're only worried about functionality in Pebble used by CockroachDB. RocksDB has a huge number of features that sometimes have bugs due to subtle interactions. There is a very stable subset of RocksDB: the configuration and specific API usage patterns used internally by Facebook. That precise combination has seen extreme testing. But that isn't the subset of RocksDB used by CockroachDB. I would guess that the most significant testing of the subset of RocksDB used by CockroachDB is the testing we do at Cockroach Labs. Now that testing is being directed at Pebble along with the Pebble-specific testing detailed in the post.
> For example, it's possible the filesystem had synced some of the buffered data to disk, but not all of it. There's no guarantee about what buffered data was synced to disk. All you know is that some, all, or none of it made it to disk.
The filesystem does provide guarantees when you use fsync() and fdatasync(). Postgres relies on these guarantees. So does RocksDB. Pebble's usage of fsync/fdatasync mirrors RocksDB's. Our crash testing is not testing the filesystem guarantees, only that we're correctly using fsync/fdatasync (which is hard enough to get right).
This is a really hard question to answer because there are so many different scenarios to test. Even if you restrict the question to KV operations a simple answer isn't forthcoming. For example, if you want a distributed system with the fastest writes, then Cassandra is your best choice. Oh, but you better pay a lot of attention to all of the caveats regarding tombstones and node outages and overwriting data.
Overall, performance has improved a lot over the past year, but we still have a lot of work to do. Some of the high-contention scenarios that caused poor performance in the Jepsen tests have seen a huge improvement (10-100x), though admittedly we were starting from a poor position.
So far we've performed the most significant comparisons against Cassandra and MongoDB. In terms of performance Cassandra > MongoDB > CockroachDB, but note that most (all?) benchmarks test idealized situations. YCSB tests ideal write scenarios where data is never overwritten or deleted and data correctness is easily achieved because there are no node outages.
Unsurprisingly, PostgreSQL will outperform a single-node Cockroach cluster for simple operations. The internal target was to be within 2x of PostgreSQL performance for simple INSERT, UPDATE, DELETE and SELECT statements. We were at that target earlier in the year, but a quick check reveals that we've slipped a bit.
Sorry I don't have anything more concrete to offer at this time. I realize I've been saying "we're working on it" for too long. Really, we are.
CockroachDB also stores SQL metadata inside of the KV store, but that metadata is also gossiped around the cluster (i.e. it is replicated to every node) so that SQL execution almost never has to read it from the KV store. Handling changes to this SQL metadata is challenging and required a design [1] that is unable to take advantage of the easy correctness of simply doing reads from the KV store on every operation.
[1] https://github.com/cockroachdb/cockroach/blob/master/docs/RF...
Re joins: we have top men working on it right now. Joins are definitely an interesting problem for a distributed database.