News for hackers. Stuff that matters.
Re:CockroachDB beta-20161013 by hinkley on Thursday February 16, 2017 @12:20PM (#13661697)
I'm a little confused about how read speed is 2x slower than write speed. With respect to 'correctness', you're drifting into pyrrhic victory or 'not even wrong' territory at that point.

When there are basic expectations of behavior that aren't being met, many of us would reject the idea that this code is 'correct'.

Re:CockroachDB beta-20161013 by arjunnarayan on Thursday February 16, 2017 @01:56PM (#13662498)
[Disclaimer: CockroachDB engineer here, working on performance and benchmarking]

IIRC, in the case that aphyr refers to for these specific numbers, the reads are scans that span multiple shards[1], while the writes are writes to single shards.

[1] even though aphyr says it's just a hundred rows, the tables are split into multiple shards because aphyr in this case was specifically testing our correctness in multi-shard contention scenarios. In production you wouldn't have multi-shard reads crop up until you were doing scans for tables that were hundreds of thousands of rows in size[2]. It's easier to picture if you think of this performance speed difference in the scenario where you are doing full table scans spanning multiple shards on multiple computers while the underlying rows are being rapidly mutated by contending write transactions. The transactions get constantly aborted to avoid giving a non-serializable result, and performance is suffering. We agree that the numbers in this contention scenario are too low, and we are actively working on high-contention performance (and performance in general) leading up to our 1.0 release[3].

[2] Specifically, we break up into multiple shards when a single shard exceeds 64mb in size.

[3] You can follow along on one of the PRs that address this specific performance issue here: https://github.com/cockroachdb/cockroach/pull/13501

Re:CockroachDB beta-20161013 by hinkley on Thursday February 16, 2017 @03:10PM (#13663143)
Oh, I see. That makes sense, thank you.

The benchmark isn't quite apples to apples. I'm used to people doing put/get benchmarks, not put/search workflows. Merging results from multiple machines is nothing to sneeze at, especially if you have replication between nodes.

I worked for a search engine dotbomb, and I had a constant worry that the core engineering team had not solved a very similar problem. But since the funding round failed I'll never know for sure what they had planned.

Re:CockroachDB beta-20161013 by grogers on Thursday February 16, 2017 @08:26PM (#13665180)
Are read-only transactions handled without contention? I understand that a read-write transaction needs to be aborted if a later conflicting write is committed. But read-only transactions can view a snapshot of what was committed before they begin - you trade a little bit of latency for being conflict free. It doesn't even have to be the default as long as you offer it as an option, you might want this if you know you are reading hot keys, or as a fallback if you've been aborted X times.

This was IMO the big innovation of true time in spanner - contention free reads at either now, or at some time in the past.

Re:CockroachDB beta-20161013 by tschottdorf on Friday February 17, 2017 @03:40AM (#13666842)
Yes, if your timestamp is far enough in the past (this is determined by the maximum clock offset configured for the cluster), and if you don't collide with writes of a long-running transaction (which may run at a timestamp that could influence what you read), you will not have to worry about contention. What you're suggesting are definitely options for reading consistent, if slightly out of date, information from a hot keyspace. There's a blog post on time travel queries, which conveniently expose the required functionality as specified by the SQL standard.
Re:CockroachDB beta-20161013 by aphyr on Thursday February 16, 2017 @12:28PM (#13661781)
As I mentioned in the article's introductory paragraphs, where Spanner forces a minimum latency on writes to ensure consistency, CockroachDB pushes that latency to reads which contend with a write.
Re:CockroachDB beta-20161013 by qaq on Thursday February 16, 2017 @12:48PM (#13661958)
Latency is one thing throughput is a different animal 2 reads/second per m3.large node that really forces one to think there are some severe architectural issues.
Re:CockroachDB beta-20161013 by hinkley on Thursday February 16, 2017 @02:58PM (#13663046)
You're not wrong. Whoever downvoted you is pushing an agenda and I'm not happy about it.

Reads per second is ultimately a measure of how many operations can be retired per second, not how long the operation takes. Anyone who has spent fifteen minutes learning about capacity planning should know that. 5 reads per second doesn't mean each read takes 200ms. It might mean 1 second per read spread across 5 threads of execution, or 3 seconds per read across 15 workers.

Re:CockroachDB beta-20161013 by stickfigure on Thursday February 16, 2017 @03:19PM (#13663209)
The parent poster (and many others in this thread) are assuming that performance under this deliberately pathological test is reflective of performance in the real world.

Optimistic locking systems inherently perform poorly under contention. But they also perform better than pessimistic concurrency systems overall because in the real world we design applications to avoid contention.

As an example, the Google App Engine datastore runs zillions of QPS across petabytes of data in a massive distributed cluster. But if you build an app that does nothing but mutate a single piece of state over and over, you'll top out at a couple transactions per second. This is painful if you're trying to build a simple counter, but with minimal care you can build a system that scales to any sized dataset and traffic volume.

Re:CockroachDB beta-20161013 by hinkley on Thursday February 16, 2017 @03:32PM (#13663322)
So the benchmark is only informative from the standpoint of determining which records should be split to avoid concurrent writes.

For instance you wouldn't expect a single user to make 5 comments or upvotes per second so storing data about recent activity with the user isn't a bottleneck that you need to design for. Storing data about responses with an item might also be okay as long as you don't plan to be HN or Reddit (Github, for example, would be just fine). But if you want to track activity globally (eg, managing the watch list notifications in github), you will need to design around that number.

Re:CockroachDB beta-20161013 by stickfigure on Thursday February 16, 2017 @03:53PM (#13663485)
Aphyr's test is not a benchmark. It is a test of database correctness under a carefully constructed set of pathological circumstances. It cannot be used to infer real-world performance behavior.

Yes, the general advice for users of the GAE datastore is to build Entity Groups around the data for a single user. That isn't absolute though, and it doesn't cause problems for watch lists; the Watch can be part of the User's EG rather than the Issue's EG. Or it can be its own EG. In practice this doesn't require as much consideration as you probably imagine.

Re:CockroachDB beta-20161013 by qaq on Thursday February 16, 2017 @03:37PM (#13663353)
Regardless of locking strategy that throughput is abysmal. None would complain about high latency.
Re:CockroachDB beta-20161013 by erichocean on Thursday February 16, 2017 @02:53PM (#13662997)
> I'm a little confused about how read speed is 2x slower than write speed.

FWIW, you're also describing Cassandra and it seems to do fine in the marketplace.

Re:CockroachDB beta-20161013 by phamilton on Friday February 17, 2017 @02:22AM (#13666584)
Also describing random read/write disk benchmarks.