News for hackers. Stuff that matters.
Re:Call Me Maybe: MongoDB Stale Reads by jxf on Tuesday April 21, 2015 @09:03PM (#9418099)
That island of safety is a rapidly shrinking one. Jepsen covers more ground with each new post from Kyle. :)
Re:Call Me Maybe: MongoDB Stale Reads by aidanhs on Tuesday April 21, 2015 @09:17PM (#9418143)
Has there been any CP system passing a call-me-maybe test first time apart from Zookeeper [1]?

[1] https://aphyr.com/posts/291-call-me-maybe-zookeeper

Re:Call Me Maybe: MongoDB Stale Reads by josephg on Tuesday April 21, 2015 @09:50PM (#9418255)
FoundationDB. Kyle didn't bother running Jepsen against FDB because foundationdb's internal testing was much more rigorous that Jepsen. The foundationdb team ran it themselves and it passed with flying colors:

http://blog.foundationdb.com/call-me-maybe-foundationdb-vs-j...

Sadly, fdb has been bought by Apple[1], and you can't download it anymore. I sincerely hope foundationdb gets opensourced or something.

[1] http://techcrunch.com/2015/03/24/apple-acquires-durable-data...

Re:Call Me Maybe: MongoDB Stale Reads by takeda on Wednesday April 22, 2015 @01:54AM (#9418831)
I believe the real reason was that FDB was not open sourced.

Jepsen tests cannot be used to prove the system is safe, but to prove it isn't.

It looks like very often he looks into source code in order to figure out how the system operates, that way he can find weaknesses and write test to his testing framework to demonstrate the issue.

I wouldn't trust any company that uses Jepsen to show that their product is safe.

[deleted]
Re:Call Me Maybe: MongoDB Stale Reads by lobster_johnson on Tuesday April 21, 2015 @10:37PM (#9418394)
He has said [1] that each test takes literally months to do. I am not surprised that he picks the most popular databases, given the amount of work.

[1] https://github.com/rethinkdb/rethinkdb/issues/1493#issuecomm...

Re:Call Me Maybe: MongoDB Stale Reads by timmaxw on Tuesday April 21, 2015 @10:33PM (#9418381)
Actually, look at some of the things that @aphyr has tweeted about FoundationDB:

https://twitter.com/aphyr/status/542792308492484608

https://twitter.com/obfuscurity/status/405016890306985984

Re:Call Me Maybe: MongoDB Stale Reads by codahale on Tuesday April 21, 2015 @09:27PM (#9418176)
Re:Call Me Maybe: MongoDB Stale Reads by tracker1 on Tuesday April 21, 2015 @09:35PM (#9418210)
Postgres can survive a network partition? I wasn't aware that master-master or sharding+replication was in the box yet?
Re:Call Me Maybe: MongoDB Stale Reads by lobster_johnson on Tuesday April 21, 2015 @10:40PM (#9418409)
Did you read the article? It's not about PostgreSQL in a distributed setup:

> Even though the Postgres server is always consistent, the distributed system composed of the server and client together may not be consistent. It’s possible for the client and server to disagree about whether or not a transaction took place.

Re:Call Me Maybe: MongoDB Stale Reads by jeffdavis on Wednesday April 22, 2015 @02:13AM (#9418865)
To me that's not surprising. If the client connection drops, you might not know whether the transaction committed or not... the only way to know is to reconnect and inspect to see what happened.

If you want to avoid that kind of problem, use 2PC.

Do you (or the author) see this as a bug, or just something that might surprise people who haven't thought through the guarantees?

Re:Call Me Maybe: MongoDB Stale Reads by cjbprime on Wednesday April 22, 2015 @09:45AM (#9420330)
It looks like you didn't read the article either -- it describes how Postgres is using 2PC, but 2PC doesn't stop the server from committed something that the client is unaware of through missing the final ACK.

Quoting the article:

> The 2PC protocol says that we must wait for the acknowledgement message to arrive in order to decide the outcome. If it doesn’t arrive, 2PC deadlocks. It’s not a partition-tolerant protocol. Waiting forever isn’t realistic for real systems, so at some point the client will time out and declare an error occurred. The commit protocol is now in an indeterminate state.

Re:Call Me Maybe: MongoDB Stale Reads by jeffdavis on Wednesday April 22, 2015 @12:08PM (#9421408)
What is the significance of that though? As you say, it's not partition-tolerant, so while partitioned, the system is down. As soon as the network issue is resolved, you can determine the state of your transaction.

It would be foolish for a client to issue a COMMIT and then assume the transaction aborted because of a connection drop. The client should wait until the connection can be reestablished and determine the real transaction state before making a decision based on it.

It's the same issue as with a power failure during fsync. The durability of that transaction is indeterminate, but it doesn't matter because the system is down. Before the system comes back, it will go through recovery, and either find the commit record or not, thus getting back to a determinate state.

Re:Call Me Maybe: MongoDB Stale Reads by ddlatham on Tuesday April 21, 2015 @11:10PM (#9418470)
HBase. With the caveats that it's piggy backing off the success of Zookeeper, and the tests were not run by Kyle himself:

https://eng.yammer.com/call-me-maybe-hbase/

Re:Call Me Maybe: MongoDB Stale Reads by yobennett on Wednesday April 22, 2015 @01:06AM (#9418743)
Note that Robert Yokota's addendum[1] points out that HBase "cannot achieve both consistency and availability." His earlier results did not take into account that HBase clients continuously retry failed ops. (According to Nicolas Liochon, upon its death a server's regions are moved to another node.) Failures start rolling in once network partition(s) extend beyond the configured timeout. Kyle [2] was not impressed:

"During the network partition, no requests are successful" is not the best result for a CP system, IMO."

HBase should provide partial availability in the face partitions.

[1] http://eng.yammer.com/call-me-maybe-hbase-addendum/

[2] https://twitter.com/aphyr/status/509841011816665088

Re:Call Me Maybe: MongoDB Stale Reads by krenoten on Wednesday April 22, 2015 @09:26AM (#9420189)
No system can achieve 100% consistency and 100% availability in the presence of partitions. It's kind of wacky that Aphyr compared those replicated consensus tools and eventually consistent stores with HBase. HBase does not use a consensus protocol for replication, it uses HDFS. HBase is not eventually consistent. There is a single authoritative server for reads and writes of a lexicographic range of keys (a region) which writes immutable store files and a WAL to HDFS. Partial availability may be achievable for reads with a significant amount of effort and latency and limiting of total cluster size by allowing non-authoritative regionservers to read the HDFS WAL + Storefiles, but this really isn't realistic. I've personally been burned by the client retry thing though, and there's not really a better solution when you consider the types of workloads HBase is actually used for and it's incredibly variable latency (it aims only for consistency and very high AVERAGE throughput, at the cost of extremely high HIGHEST latency). One solution here could be the use of a configurable filesystem queue for clients. This is how you build resilient high-throughput pipelines. HBase is used in some places for OLTP, but only when the readload is very very low. So the effort to make reads more highly available would be in vain.
Re:Call Me Maybe: MongoDB Stale Reads by ddlatham on Wednesday April 22, 2015 @09:28AM (#9420208)
No database can "achieve both consistency and availability" during a partition.

Also, if you follow the rest of the twitter conversation you may realize, as they did, that only requests to the minority partition are unsuccessful - which is exactly what you want from CP.

Re:Call Me Maybe: MongoDB Stale Reads by platz on Tuesday April 21, 2015 @11:47PM (#9418572)
I wonder why the jeppsen test for Cassandra was deleted (or moved) https://aphyr.com/posts/294-call-me-maybe-cassandra/
Re:Call Me Maybe: MongoDB Stale Reads by edmccard on Wednesday April 22, 2015 @12:04AM (#9418610)
That link works for me--what's supposed to be missing?
Re:Call Me Maybe: MongoDB Stale Reads by platz on Wednesday April 22, 2015 @12:09AM (#9418618)
My link works, but many of the links in the article itself no longer exist
Re:Call Me Maybe: MongoDB Stale Reads by bbgm on Wednesday April 22, 2015 @12:21AM (#9418643)
From the looks of it, they're now under a branch called 'old'

https://github.com/aphyr/jepsen/tree/old