News for hackers. Stuff that matters.
Re:Call Me Maybe: MongoDB Stale Reads by jxf on Tuesday April 21, 2015 @08:30PM (#9417996)
The most interesting lessons from the Jepsen series:

* You should never trust, and always verify, the claims made by database manufacturers.

* Especially when those claims relate to data integrity.

* Super-especially when every safety level provided by the manufacturer that includes the word "SAFE" is actually unsafe.

Re:Call Me Maybe: MongoDB Stale Reads by DenisM on Tuesday April 21, 2015 @08:59PM (#9418087)
It seems easier to me to just go with a reputable database vendor.
Re:Call Me Maybe: MongoDB Stale Reads by jxf on Tuesday April 21, 2015 @09:03PM (#9418099)
That island of safety is a rapidly shrinking one. Jepsen covers more ground with each new post from Kyle. :)
Re:Call Me Maybe: MongoDB Stale Reads by aidanhs on Tuesday April 21, 2015 @09:17PM (#9418143)
Has there been any CP system passing a call-me-maybe test first time apart from Zookeeper [1]?

[1] https://aphyr.com/posts/291-call-me-maybe-zookeeper

Re:Call Me Maybe: MongoDB Stale Reads by josephg on Tuesday April 21, 2015 @09:50PM (#9418255)
FoundationDB. Kyle didn't bother running Jepsen against FDB because foundationdb's internal testing was much more rigorous that Jepsen. The foundationdb team ran it themselves and it passed with flying colors:

http://blog.foundationdb.com/call-me-maybe-foundationdb-vs-j...

Sadly, fdb has been bought by Apple[1], and you can't download it anymore. I sincerely hope foundationdb gets opensourced or something.

[1] http://techcrunch.com/2015/03/24/apple-acquires-durable-data...

Re:Call Me Maybe: MongoDB Stale Reads by takeda on Wednesday April 22, 2015 @01:54AM (#9418831)
I believe the real reason was that FDB was not open sourced.

Jepsen tests cannot be used to prove the system is safe, but to prove it isn't.

It looks like very often he looks into source code in order to figure out how the system operates, that way he can find weaknesses and write test to his testing framework to demonstrate the issue.

I wouldn't trust any company that uses Jepsen to show that their product is safe.

[deleted]
Re:Call Me Maybe: MongoDB Stale Reads by lobster_johnson on Tuesday April 21, 2015 @10:37PM (#9418394)
He has said [1] that each test takes literally months to do. I am not surprised that he picks the most popular databases, given the amount of work.

[1] https://github.com/rethinkdb/rethinkdb/issues/1493#issuecomm...

Re:Call Me Maybe: MongoDB Stale Reads by timmaxw on Tuesday April 21, 2015 @10:33PM (#9418381)
Actually, look at some of the things that @aphyr has tweeted about FoundationDB:

https://twitter.com/aphyr/status/542792308492484608

https://twitter.com/obfuscurity/status/405016890306985984

Re:Call Me Maybe: MongoDB Stale Reads by codahale on Tuesday April 21, 2015 @09:27PM (#9418176)
Re:Call Me Maybe: MongoDB Stale Reads by tracker1 on Tuesday April 21, 2015 @09:35PM (#9418210)
Postgres can survive a network partition? I wasn't aware that master-master or sharding+replication was in the box yet?
Re:Call Me Maybe: MongoDB Stale Reads by lobster_johnson on Tuesday April 21, 2015 @10:40PM (#9418409)
Did you read the article? It's not about PostgreSQL in a distributed setup:

> Even though the Postgres server is always consistent, the distributed system composed of the server and client together may not be consistent. It’s possible for the client and server to disagree about whether or not a transaction took place.

Re:Call Me Maybe: MongoDB Stale Reads by jeffdavis on Wednesday April 22, 2015 @02:13AM (#9418865)
To me that's not surprising. If the client connection drops, you might not know whether the transaction committed or not... the only way to know is to reconnect and inspect to see what happened.

If you want to avoid that kind of problem, use 2PC.

Do you (or the author) see this as a bug, or just something that might surprise people who haven't thought through the guarantees?

Re:Call Me Maybe: MongoDB Stale Reads by cjbprime on Wednesday April 22, 2015 @09:45AM (#9420330)
It looks like you didn't read the article either -- it describes how Postgres is using 2PC, but 2PC doesn't stop the server from committed something that the client is unaware of through missing the final ACK.

Quoting the article:

> The 2PC protocol says that we must wait for the acknowledgement message to arrive in order to decide the outcome. If it doesn’t arrive, 2PC deadlocks. It’s not a partition-tolerant protocol. Waiting forever isn’t realistic for real systems, so at some point the client will time out and declare an error occurred. The commit protocol is now in an indeterminate state.

Re:Call Me Maybe: MongoDB Stale Reads by jeffdavis on Wednesday April 22, 2015 @12:08PM (#9421408)
What is the significance of that though? As you say, it's not partition-tolerant, so while partitioned, the system is down. As soon as the network issue is resolved, you can determine the state of your transaction.

It would be foolish for a client to issue a COMMIT and then assume the transaction aborted because of a connection drop. The client should wait until the connection can be reestablished and determine the real transaction state before making a decision based on it.

It's the same issue as with a power failure during fsync. The durability of that transaction is indeterminate, but it doesn't matter because the system is down. Before the system comes back, it will go through recovery, and either find the commit record or not, thus getting back to a determinate state.

Re:Call Me Maybe: MongoDB Stale Reads by ddlatham on Tuesday April 21, 2015 @11:10PM (#9418470)
HBase. With the caveats that it's piggy backing off the success of Zookeeper, and the tests were not run by Kyle himself:

https://eng.yammer.com/call-me-maybe-hbase/

Re:Call Me Maybe: MongoDB Stale Reads by yobennett on Wednesday April 22, 2015 @01:06AM (#9418743)
Note that Robert Yokota's addendum[1] points out that HBase "cannot achieve both consistency and availability." His earlier results did not take into account that HBase clients continuously retry failed ops. (According to Nicolas Liochon, upon its death a server's regions are moved to another node.) Failures start rolling in once network partition(s) extend beyond the configured timeout. Kyle [2] was not impressed:

"During the network partition, no requests are successful" is not the best result for a CP system, IMO."

HBase should provide partial availability in the face partitions.

[1] http://eng.yammer.com/call-me-maybe-hbase-addendum/

[2] https://twitter.com/aphyr/status/509841011816665088

Re:Call Me Maybe: MongoDB Stale Reads by krenoten on Wednesday April 22, 2015 @09:26AM (#9420189)
No system can achieve 100% consistency and 100% availability in the presence of partitions. It's kind of wacky that Aphyr compared those replicated consensus tools and eventually consistent stores with HBase. HBase does not use a consensus protocol for replication, it uses HDFS. HBase is not eventually consistent. There is a single authoritative server for reads and writes of a lexicographic range of keys (a region) which writes immutable store files and a WAL to HDFS. Partial availability may be achievable for reads with a significant amount of effort and latency and limiting of total cluster size by allowing non-authoritative regionservers to read the HDFS WAL + Storefiles, but this really isn't realistic. I've personally been burned by the client retry thing though, and there's not really a better solution when you consider the types of workloads HBase is actually used for and it's incredibly variable latency (it aims only for consistency and very high AVERAGE throughput, at the cost of extremely high HIGHEST latency). One solution here could be the use of a configurable filesystem queue for clients. This is how you build resilient high-throughput pipelines. HBase is used in some places for OLTP, but only when the readload is very very low. So the effort to make reads more highly available would be in vain.
Re:Call Me Maybe: MongoDB Stale Reads by ddlatham on Wednesday April 22, 2015 @09:28AM (#9420208)
No database can "achieve both consistency and availability" during a partition.

Also, if you follow the rest of the twitter conversation you may realize, as they did, that only requests to the minority partition are unsuccessful - which is exactly what you want from CP.

Re:Call Me Maybe: MongoDB Stale Reads by platz on Tuesday April 21, 2015 @11:47PM (#9418572)
I wonder why the jeppsen test for Cassandra was deleted (or moved) https://aphyr.com/posts/294-call-me-maybe-cassandra/
Re:Call Me Maybe: MongoDB Stale Reads by edmccard on Wednesday April 22, 2015 @12:04AM (#9418610)
That link works for me--what's supposed to be missing?
Re:Call Me Maybe: MongoDB Stale Reads by platz on Wednesday April 22, 2015 @12:09AM (#9418618)
My link works, but many of the links in the article itself no longer exist
Re:Call Me Maybe: MongoDB Stale Reads by bbgm on Wednesday April 22, 2015 @12:21AM (#9418643)
From the looks of it, they're now under a branch called 'old'

https://github.com/aphyr/jepsen/tree/old

Re:Call Me Maybe: MongoDB Stale Reads by lmm on Wednesday April 22, 2015 @05:42AM (#9419320)
Is there actually that much difference between MongoDB and e.g. Cassandra? The tone of the Call Me Maybe posts about both is quite different, but the bottom line seems to be the same: use CRDTs for everything or you will lose data.
Re:Call Me Maybe: MongoDB Stale Reads by threeseed on Tuesday April 21, 2015 @08:38PM (#9418027)
Actually the broader lesson would be to assume the worst in your application layer and try and remediate/verify wherever possible.

If you look at his articles: Redis, PostgreSQL, Cassandra, ElasticSearch etc all had data consistency errors. And none of those have vendors making any claims.

It's pretty sobering to say the least.

Re:Call Me Maybe: MongoDB Stale Reads by ak4g on Tuesday April 21, 2015 @09:09PM (#9418119)
Um, this is the postgres article:

https://aphyr.com/posts/282-call-me-maybe-postgres

There were no acknowledged writes lost. The only unacked-but-successful writes resulted from a connection while a commit ack was in-flight. That doesn't qualify as a data-consistency error, it means the client has to check if the data is present after reconnecting.

But in no cases would the client reconnect to find that there were acknowledged-as-committed records that were missing or stale. In no cases would the client find that responded-as-rolled-back data was actually committed. This is very, very different than what is seen with MongoDB.

Re:Call Me Maybe: MongoDB Stale Reads by hendzen on Tuesday April 21, 2015 @09:29PM (#9418184)
I don't think the results of Aphyr's MongoDB and postgres experiments are directly comparable. In the OP, MongoDB was run in a 5 node replicated configuration. In the post you reference, the experiment was run against a single postgres node.

Furthermore, the postgres experiment only checked that no writes were lost. As Aphyr acknowledges, MongoDB did not lose any writes with "majority" write concern. The postgres experiment did not include the verification of a linearizable history of reads, which is what the bulk of the OP is about.

I'd like to see a similar experiment run against a replicated postgres configuration with auto-failover.

Re:Call Me Maybe: MongoDB Stale Reads by tracker1 on Tuesday April 21, 2015 @09:34PM (#9418204)
Now try it again with PostgreSQL's built in sharding or replication functionality... oh, wait.
Re:Call Me Maybe: MongoDB Stale Reads by zapov on Tuesday April 21, 2015 @09:41PM (#9418227)
The parent post correctly pointed out that including Postgres in that list is misleading at least.
Re:Call Me Maybe: MongoDB Stale Reads by threeseed on Tuesday April 21, 2015 @09:55PM (#9418271)
I was talking about data consistency more broadly. "However, two writes (215 and 218) succeeded, even though they threw an exception claiming that a failure occurred". This obviously isn't ideal behaviour from the perspective of a developer but isn't necessarily not correct.

My broader point was that you need to assume the worst from your database in the application layer. Which I think you missed.

Re:Call Me Maybe: MongoDB Stale Reads by zapov on Tuesday April 21, 2015 @10:32PM (#9418377)
I read your post as FUD regarding Postgres.

That article has various issues, for example, calling Postgres commit protocol as a special case of two phase commit is not really correct. Postgres has 2pc: http://www.postgresql.org/docs/9.2/static/sql-prepare-transa... but that was not tested.

The described behavior is "expected" and "understood". Saying that "you should assume worst from your database" is not something I would ever use for describing DB with ACID semantics.

Re:Call Me Maybe: MongoDB Stale Reads by threeseed on Tuesday April 21, 2015 @11:08PM (#9418463)
I wasn't spreading FUD about anything. From the article there was issues with every databases expected behaviour. My point again was that you should expect and manage failure in your application layer. It's what sensible architecture looks like.

And ACID does NOT guarentee that you will not lose data. It is a theory not an implementation. I have lost data with both Oracle and Teradata due to bugs.

Re:Call Me Maybe: MongoDB Stale Reads by zapov on Tuesday April 21, 2015 @11:21PM (#9418500)
Well, you said that Postgres had data consistency errors (referencing Aphyr articles). This is not true (at least regarding that article).

Aphyr article about Postgres could be renamed to call-me-maybe-acid-db-over-the-network and could remain the same.

Re:Call Me Maybe: MongoDB Stale Reads by cpks on Wednesday April 22, 2015 @06:46AM (#9419500)
You misunderstand the results. PostgreSQL behaves as expected, and indeed, the only way it can behave. That's the Two General's Problem (http://en.wikipedia.org/wiki/Two_Generals%27_Problem). There is no way to solve it. PostgreSQL does as well as theoretically possible.

I understand you lost data with Oracle and Teradata.

1. Most big corporate vendors do not have systems which are very well designed. (1) The sale is made at a business level. (2) Most Oracle customers are not very tech companies, and have mixed quality employees. As a result, there is little pressure on building a robust, correct product, rather than one which meets a feature checklist. In addition, Oracle doesn't really recruit smart people (I know people who work there). It's just not very robust compared to something like PostgreSQL, which was written by Stonebraker, a legendary computer science professor and entrepreneur.

2. Still, more likely, the reason you lost data is because you didn't know what you were doing. Words like Eventual Consistency, ACID, Two-General's Problem, etc. are not just abstract. They have strict, formal meanings, and you need to understand what they do and do not guarantee. Otherwise, you will lose data again.

Re:Call Me Maybe: MongoDB Stale Reads by threeseed on Saturday April 25, 2015 @07:39PM (#9439931)
> the reason you lost data is because you didn't know what you were doing

Missed this reply and thought it was funny. I work for one of the world's largest retailers and we are one of both Oracle's and Teradata's most loved customers. We have 4 Teradata DBAs provided BY Teradata amongst a team of 20 SQL Developers. We aren't messing around.

What YOU don't seem to understand is that bugs in your database can cause data loss. ACID or Strong Consistency will not save you.

Re:Call Me Maybe: MongoDB Stale Reads by cpks on Sunday May 03, 2015 @05:12PM (#9483068)
That's not exactly the sort of team where you'd expect MIT Ph.Ds to work. It's precisely big teams of mediocre people who run into issues based on not knowing exactly what the database is doing and how it's supposed to work that lead to data corruption due to misuse. It's almost always a boring problem (e.g. retail business software). It's almost always a clunky commercial "enterprise" solution (e.g. Oracle). It's almost always a big team. I'm not sure if I even need to go into application engineers -- you put your best and brightest into the core product, and solutions typically gets those that can't quite cut it there.

Regardless, your comment was about PostgreSQL, not Oracle. Oracle is a giant piece of software written by a corporation with over 100,000 employees. Something like that is bound to have bugs, and it has bugs indeed. Data corruption with Oracle certainly happens. PostgreSQL is written by a small, ultra-elite team. It's a much smaller codebase, so an expert developer can understand how the whole system works. There's a big difference in robustness between the two.

Of course database bugs can cause corruption. I've certainly had MonogDB eat my data. But the level of robustness of different databases is very different. There are many databases which are essentially bug-free. If you're losing data with PostgreSQL, odds are you're the one losing the data, not PostgreSQL.

Re:Call Me Maybe: MongoDB Stale Reads by rdtsc on Tuesday April 21, 2015 @10:08PM (#9418307)
Actually, just use Riak. Those people know distributed bases.

https://aphyr.com/posts/285-call-me-maybe-riak

Now I hear they support consistency as well.

Re:Call Me Maybe: MongoDB Stale Reads by iconfluence on Tuesday April 21, 2015 @10:16PM (#9418337)
I think at one point the employees of Basho knew how to write distributed DBs - Riak is the most advanced AP DB from a distributed systems theory perspective. However, in recent months, their CEO, CTO and Chief Architect have left, as well as many of their prominent engineers. Worryingly, the new CTO seems content to make inane comments about "Data Gravity" [0].

[0] - http://www.kdnuggets.com/2015/03/interview-dave-mccrory-bash...

Re:Call Me Maybe: MongoDB Stale Reads by rspeer on Wednesday April 22, 2015 @01:53AM (#9418828)
I think Riak's theory is fine, but theory isn't enough. And they may have succumbed to the Osborne effect with Riak 2.0.

Here's what I mean. Think of all the nice things you expect to come out of Riak's theoretical basis -- bulletproof distributed writes, for example.

Well, the default last-write-wins writes aren't bulletproof. They clearly fail Jepsen [1]. And if you turned off last-write-wins, then you'd have to handle siblings -- and how you were supposed to handle them without introducing inconsistency was quite unspecified. Riak clients gave you no help there.

Or, as the Jepsen article says, you could use CRDTs. Before 2.0, Riak had exactly one CRDT, the counter.

Something that might be appealing to some is built-in MapReduce. On the forums, people would warn you not to actually use it unless you didn't actually want availability after all. I don't know if they ever sorted that out.

Another supposedly nice thing was Riak Search. A distributed DB with full-text search out of the box -- that sounds great, right? But there were two things called Riak Search, and the first one just plain didn't work. They deprecated it before it had a replacement, but the replacement was supposed to come in 2.0.

So, Osborne effect. When people gradually discovered that Riak 1.x was bad, and the response was "but Riak 2.0 will be great!", that's a great reason not to use 1.x. 2.0 took a very, very long time, enough time for customers and potential customers, including us, to find other solutions.

[1] https://aphyr.com/posts/285-call-me-maybe-riak

Re:Call Me Maybe: MongoDB Stale Reads by siculars on Wednesday April 22, 2015 @08:15PM (#9424342)
Ya, you have to handle siblings if you don't use LWW. If you would rather Riak execute a pre-defined merge strategy, use Riak's CRDT features.

The original Riak Search in vs 1.x was replaced with integrated Solr in Riak 2.x, and it works.

I'd love to hear more about your use case. Contact info in profile. I'm this username in all the usual suspects.

Disclaimer: I work at Basho.

Re:Call Me Maybe: MongoDB Stale Reads by biggest_lou on Wednesday April 22, 2015 @09:29AM (#9420216)
FWIW I did write a bunch of documentation on sibling resolution in Riak: http://docs.basho.com/riak/latest/dev/using/conflict-resolut...
Re:Call Me Maybe: MongoDB Stale Reads by rspeer on Wednesday April 22, 2015 @11:26AM (#9421108)
So it looks like the conflict resolution strategies being recommended are "pick one arbitrarily", or in an advanced section, "keep the longer list"?

Not that it matters to me anymore, but it does sound like having all data in CRDTs is the only way to pass Jepsen. (Or to have your data be immutable, in which case Jepsen doesn't apply.)

Re:Call Me Maybe: MongoDB Stale Reads by scurvy on Tuesday April 21, 2015 @10:39PM (#9418401)
Indeed. We're evaluating Riak CS and Swift. In a vacuum, I'd choose Riak CS any day. Knowing all of the drama at Basho, we're in a holding pattern at best and leaning (reluctantly) towards Swift.

Basho needs to sell damn fast or just call it a day and open source the enterprise version.

Re:Call Me Maybe: MongoDB Stale Reads by rdtsc on Tuesday April 21, 2015 @10:57PM (#9418441)
This is kind of surprising to hear. I had no idea.

Well in that case have you heard about LeoFS? Wonder if it overlaps with any of the Riak CS features for you.

http://leo-project.net/leofs/

Re:Call Me Maybe: MongoDB Stale Reads by siculars on Wednesday April 22, 2015 @08:18PM (#9424353)
The drama has died down. Drama is generally isolated to pitched battles amongst engineers for feature implementations ;)

Contact info in profile, this username at all the usual suspects.

Disclaimer: I work for Basho.

Re:Call Me Maybe: MongoDB Stale Reads by jodah on Tuesday April 21, 2015 @10:56PM (#9418438)
Drama aside, Riak and Swift are not in the same league.
Re:Call Me Maybe: MongoDB Stale Reads by scurvy on Thursday April 23, 2015 @02:51PM (#9428955)
Riak CS is. I said Riak CS, not Riak.
Re:Call Me Maybe: MongoDB Stale Reads by biokoda on Wednesday April 22, 2015 @01:31AM (#9418790)
That was a year ago and they seem to be doing fine since.
Re:Call Me Maybe: MongoDB Stale Reads by kokey on Wednesday April 22, 2015 @07:44AM (#9419695)
I had a bad experience with riak v2. I think something went quite wrong in the process when going from v1 to v2, especially around Riak Search.
Re:Call Me Maybe: MongoDB Stale Reads by siculars on Wednesday April 22, 2015 @08:21PM (#9424358)
Riak Search in v1.x and Riak Search in v2.x are completely different. Riak 2.x tightly integrats Solr.

If you wanna talk about it I'm this username at all the usual online places.

Disclaimer: I work for Basho

Re:Call Me Maybe: MongoDB Stale Reads by ChuckMcM on Tuesday April 21, 2015 @10:37PM (#9418393)
oooh, that is scary.
Re:Call Me Maybe: MongoDB Stale Reads by siculars on Wednesday April 22, 2015 @08:23PM (#9424364)
We now offer a strong consistency option.

I'm this username in the usual places online if you wanna talk about it.

Disclaimer: I work for Basho

Re:Call Me Maybe: MongoDB Stale Reads by chatman on Tuesday April 21, 2015 @11:02PM (#9418451)
Apache Solr's jepsen tests have had very good results.