> In and eventually consistent system, the programmer must consider what to do in the case of data failure.
Eventually consistency has nothing to do with data failure; the real-world effect of eventually consistency is stale data, which you can't generally tell is stale.
> When building a user authentication system, who better to decide what to do when you can't access the user database than the person writing the user auth system?
I think you're talking about availability, not (eventual) consistency.
> Also, it means that joins must be done in your application.
There is nothing preventing you from doing joins in a strongly consistent database. Google Spanner supports JOINs for example [1].
> In an eventually consistent system, you know if you created a table scan, because you have to pull back a full table's worth of data
This also has nothing to do with eventual consistency.
[1] https://cloud.google.com/spanner/docs/query-syntax#join-hint...
> Eventually consistency has nothing to do with data failure; the real-world effect of eventually consistency is stale data, which you can't generally tell is stale.
You're right, that is the highly available distributed part, but it goes along with eventual consistency because if you plan for dealing with stale data then those same concepts protect you from unavailable data.
> I think you're talking about availability, not (eventual) consistency.
Again, the two go hand in hand. Deal with stale data and you've dealt with unavailable data.
> There is nothing preventing you from doing joins in a strongly consistent database.
You misread -- I was saying that with a multi-master key-value system you have to do joins in the app.
> This also has nothing to do with eventual consistency.
Same, I was talking about Dynamo-like systems.
It looks like you’re conflating a well-defined term in distributed systems with properties of particular database implementations that have nothing to do with eventual consistency.
Nothing in your reply is a necessary property of eventually consistent systems, although some eventually consistent systems may be built with those properties. That is quite a huge difference when we’re in the context of the blog post being discussed.
The grayness of your reply should suggest that perhaps you’re misusing the term, rather than everyone else being “confused”.
Unfortunately, I think you've come to the opposite conclusion in this case. When jedberg writes the code for a system backed by an eventually-consistent data store, he asks "What happens if this value is stale and how can I structure the code to cope with that in a sane and reasonable way?"
But when $Random_Dev writes the code for such a system, he asks some combination of "What does eventually consistent mean?", "What do you mean I won't get back the right data?", and "Isn't that the DBA's problem?"
The core issue being that the average dev will punctuate this line of questioning with things that shift the blame elsewhere; if not the DBA, then "Well, it's good enough for [INSERT_TRENDY_COMPANY_HERE] [ reddit? ;) ], so I think we'll be OK!" or "I don't think that will ever happen to us, our reads get consistent quickly enough". Just whatever will punt the issue down the road.
To be honest, I believe that this impulse is the impetus behind the rise in NoSQL in the first place. Devs resent DBAs and SQL. They, very simply, want a place to shove their data, a place to read it out, and to not know anything else about the in between. I have seen many pitiful architectures that clearly turned on this consideration (and we're seeing it all over again with cloud orchestration).
Good developers will dig in and search out the meaning of the data whether the failure case is handled at the DB level or not.
Highly-structured systems like a traditional RDBMS are very good for resiliency because they require someone to put some definition and formality around the dataset to get basic functionality, and they are generally extremely careful about ensuring the things that come in and out of them are safe and accurate. That's the kind of backing that we really need, especially when there are many developers who can't be trusted to even try to understand more than the bare minimum necessary to get a positive quarterly review.
That's not what happens in practice. Just because an RDBMS has transactions doesn't mean a programmer who isn't trying is going to use them correctly. And when you do enforce constraints at the DB level, you end up duplicating all your logic in two different languages, one of which isn't very nice for programming in. E.g. I've never found database timezone handling to be anything other than harmful. The application is just better at enforcing this kind of constraints; a well-designed application could make use of transactionality, but putting the transactionality in the datastore won't make the application better at it if the dev wasn't thinking about it - and if the dev is thinking about it, they can usually do better at the application level.
Definitely true, but RDBMS provide a lot of safety features beyond transactions, like a fixed schema. Mongo et al let developers stuff anything they want anywhere, creating an abominable bastardization between a key-value store and a database. That's their "killer feature" that has resulted in wide adoption so they can't really do away with it, even if some [likely reluctant] Mongo users continue to find ways to self-enforce a schema.
It'd be one thing if Mongo was predominantly used as a staging ground for free-form, unpredictable data, but I think we all know that's not the case. People pretend like it's a replacement for a traditional RDBMS.
>a well-designed application could make use of transactionality, but putting the transactionality in the datastore won't make the application better at it if the dev wasn't thinking about it - and if the dev is thinking about it, they can usually do better at the application level.
I really disagree here. The semantics and detail involved in _actually_ making sure a datum is securely stored, including a high likelihood of being recoverable in the case of a crash, is very complex in any non-trivial system. It's dismissive and naive to pretend that your average application developer, who generally doesn't even know what `fsync()` is, can do a better job at this than a real database, which will include advanced consistency features like intent logs.
Their real killer feature was proper multi-master clustering - being able to seamlessly add and remove nodes from your storage cluster - in a free product. Traditional RDBMS don't do that.
> The semantics and detail involved in _actually_ making sure a datum is securely stored, including a high likelihood of being recoverable in the case of a crash, is very complex in any non-trivial system. It's dismissive and naive to pretend that your average application developer, who generally doesn't even know what `fsync()` is, can do a better job at this than a real database, which will include advanced consistency features like intent logs.
Intent logs are necessary precisely because the database table model is a poor fit for what actually happens - even databases themselves now spend more effort on converting back and forth to the SQL model than on actual storage or query implementation. A developer with the skills to do it right does need a library of certain capabilities, but the RDBMS is not a good interface for supplying them with those capabilities. See e.g. https://www.confluent.io/blog/turning-the-database-inside-ou... .
In C# for instance using the standard Mongo driver, you're dealing with strongly typed collections and the compiler is keeping you from mistakingly going outside of the "schema".
I don't know how good the other .Net drivers are for other Nosql databases.
We basically come from opposite viewpoints -- I think most devs will do the right thing, you think most will do the wrong thing.
Honestly, you're probably right and I'm probably wrong, but I like to hope that I will always work in a place where I'm right. :)
(It was Netflix BTW, not reddit. For the most part reddit used strongly consistent databases (Postgres), but we did use some eventual consistency in the caching layer).
If you use it in the wrong places, then that almost certainly means increases complexity where you don't necessarily have to. Whether or not that results other positive behaviours seems a bit beside the point to me.
If you managed a team of rock climbers, you could say "I prefer it when my rock climbers don't use safety ropes. It improves their climbing, because if they make a mistake, they'll fall to their death". It could be 100% true, but it doesn't mean you shouldn't simply be picking the right tools for the job.
This comes down to culture. I wish I knew how to create a culture where devs do the right thing, from a culture where they ¯\_(ツ)_/¯.
Maybe you need the very clearly Netflix model of "Fire every ¯\_(ツ)_/¯"
Outside of AWS, Google and Azure I see few cloud services that define their consistency models. And support always says: we are high availability and always consistent.
Same goes for distributed software, few projects define their consistency model, and even fewer honor it.
That despite the fact that I changed my address when I moved (says so in the TxDPS record, running on a "legacy" system), which got sent to the toll agency when I changed it. Of course somewhere in there the toll agency lost this information and never bothered to check the TxDPS record until I pointed out they were repeatedly sending notices to the wrong address.
The point being that, this is only one of many problems they have had with their newfangled toll system because sometimes "eventually consistent" actually means "we threw the update away because a newer one came in, so we lost a critical piece of information.". Critical pieces of information like, people actually paying their tolls for example.
http://www.texasturf.org/2012-06-01-03-09-30/latest-news/484... http://story.kxan.com/txtag-troubles/ https://www.reddit.com/r/Austin/comments/78wp9y/txtag_paymen...
I think I can understand some of your point. Trust should be engineered into the system, not assumed. That is a lot of work and not easy, though. (I think this ultimately lands on the block chain idea of the world. I'm fairly ignorant there.)
Strong consistency just means that you can expect any accesses you performed to be consistent in time with each other, right? To that end, it is nice not having to effectively do a "loopUntilChangeReflected" after every write and checking to see if it is safe to do the next thing. Which you have to do in an eventually consistent world.
If you have to do this, you probably shouldn't be using an eventually consistent system. Eventually consistent systems should only be deployed when a stale read is harmless. For example, it probably isn't a big deal if something reads "5 upvotes" instead of "7 upvotes" before a change fully propagates out.
Wrapping every write in a "block to ensure consistency" loop is just a lossy, slow, terrible and dangerous way to recapture a tiny amount of the consistency available to those who use a system that suits their needs.
I know that in the real world, we don't often get to make these decisions, or there may be one-off edge cases (though really just put these on a system that's safe for them). But I just want to point out to other readers that if you catch yourself doing this, you just need to use a datastore that fits your application's use case.
Yes, even if you just read an article in TechCrunch about the Hippest Coolest Hottest Big-Dataiest 2.0 3.0 4.0 5.0 Data Storage Project That Makes Google and Facebooks' Mouths Water Hadoop-Daboop Machine, That Only Cool People Use, and No Uncool People Or Trump Supporters Of Any Kind Allowed.
Read that article, and then go use Postgres.
I'd still argue that even reasoning about all writes in an, "eventually this is true for everyone else," fashion is less than straight forward.
To that end, components should all be able to assume what they do is strongly consistent. If possible, assume everyone else is behind?
This is in contrast to a bank which should always operate only when there is no partition.
That is, how is it simpler to assume that the data you get back could be wrong? Now, anyone involved with the system seems to have to know how everyone involved in the system interprets results.
This meshes with the articles assertion: strong consistency leads to fewer bugs because fewer possible states exist at application level.
If the "play movie" service calls the auth service and says "should I let this play", the auth service returns "authorized" or didn't give a response or said "maybe authorized" or whatever. The "play movie" service makes the business decision that makes sense for playing a movie, in this case, "go ahead".
But the billing service maybe says "I'm only gonna let you in with a confirmed yes". So it's possible the billing service will give you an error in the same conditions that the movie service does not, but those are business decisions instead of technical ones, which is good.
I'd want strong semantics on, "what is allowed with an authenticated connection with an authentication that is X old?". Preferably, I'd want a somewhat knowable centralized list of the answers. These are not software concerns, but business ones. Right?
Netflix did for a while at least, may still do it. Worst case is that someone didn't pay for the month. The partitions happen rarely enough that it is better to give the user a good experience then not let a legit user in because of a network partition.
But the nice part is that this was a business decision, not a database decision.
Allow them to watch a movie, but don't show billing details.
In theater parlance, people in the theater are assumed authorized to be there by virtue of the entrance requiring authorization.
Right?
Similarly, this had little to do with consistency. Any client that makes a logout should assume the connections it has are no longer valid. To assume otherwise seems odd. Worse, to encourage relying on it.
Worst case would be a renegade client that constantly used the open door, no?
I'm envisioning something on the backend equivalent to:
OnGetVideoChunkRequest: if Authorized(User) or IsDown(AuthSystem) then SendVideoChunk
Now, checking a password in the system should be strongly consistent. In that I should be able to update my password and immediately use the updated password. Any other behavior will surprise me.
For purposes of discussing fail open, how that logic gets translated during implementation doesn't matter. Authorized(User) might be expiring tokens passed in via cookie. IsDown(AuthSystem) could just check some global flag that gets set by some watchdog service on a AuthSystem heartbeat. It could be something else. It doesn't matter because it's not relevant.
Having a direct connection to the AuthSystem or not is an implementation detail, not something that changes the idea "Netflix should fail open".
Similarly, anywhere you are doing idempotent guarantees, strong consistency is vital. Same for customer interaction. Reads following writes that a user made should almost certainly strive for strong consistency. If only to avoid confused customers.
Writes the system makes, though? Expect those to just fail in ways that will leave things inconsistent and constantly check for them.
But my point was eventually consistent systems, in general, force you to think about these things earlier on. Of course there are exceptions in both directions.
This is because, not only did we develop an edict against stored procedures, but we then developed an edict that everything must be done using the lowest consistency model possible. We ended up settling on session consistency. TBH we discovered an algorithm that didn't even require session consistency, but the latency would have been to great. Again, no particular analysis or rationale for the edict, just idealism. The end result was a system that, for each update, required six background jobs to be launched, which each launched their own cascading jobs that patched up the data. And our records all had to be peppered with version fields and updated-by-job fields and version-of-foreign-key fields that these fixup jobs had to test against to make sure they're not operating against stale data or running over each other, such that this crap actually outweighs the actual data in the records. And these background jobs manage state in Table Storage and transfer messages via Service Bus, so two more SPOFs not to mention all the extra compute resource cost. It's a huge mess of a jenga puzzle that's taken months to implement and code review and test, and now even six months later we're still finding edge cases that aren't handled correctly (or are they? you have to constantly go through from the start to really understand it). In addition, since things are never guaranteed to be in a consistent state, that means we could have foreign keys that don't reference anything, or circular graphs where cycles should be disallowed, or whatever else at any moment in time, and so we have to fudge this stuff when presenting data to the user, and it reduces our ability to do any sort of meaningful statistics on our data. To remove 13 lines of javascript. It's just a mess.
And then what happened is a couple new requirements came in that couldn't be reasonably handled in a lazy fashion, so we ended up using stored procs for that logic anyway.
Now, to be clear, the resulting system is strictly more scalable than the stored proc based one. One could in theory just throw more machines at any load and it'll continue to work. So there's that. But a) the load beyond which the stored proc system could scale is about 3 orders of magnitude higher than even our optimistic usage estimates in the foreseeable future, and b) the lazy algorithm requires so much additional compute overhead that, while infinitely scalable, would we even be able to afford it anymore?
So for my money, give me a strongly consistent system by default. I don't even particularly understand the argument given above. With a strongly consistent system, you can still choose to do things like join in the client if there's a reason for it. But going the opposite way and forcing consistency to be unassumed everywhere produces the mess we're dwelling in now.
Now, certainly there are cases where eventual consistency is better: high-write-volume on not-very-relational data where point-in-time accuracy isn't super business-critical is where it typically shines. At the "edges" of your system, essentially. But I do want to point out some of the pain that can be encountered if it's chosen for the wrong use case, and make sure that anyone who does use it in their core application logic knows what they're in for, and puts some more thought into whether it's what they really want.
That sounds bad. The way you're supposed to do it is have the initial data be immutable and compute the downstream data separately, but into its own places.
> So for my money, give me a strongly consistent system by default. I don't even particularly understand the argument given above. With a strongly consistent system, you can still choose to do things like join in the client if there's a reason for it. But going the opposite way and forcing consistency to be unassumed everywhere produces the mess we're dwelling in now.
I'd say transactions should never be the default, because a database transaction that isn't a semantic transaction is worse than useless. If the application programmer hasn't thought about transactionality and synchronization, it's much better for that to show up as not having any transactions - you can fix that, by adding transactions where you need them - than to have transactions that don't mean anything, and the data ending up inconsistent even though no transactions ever overlapped.
There is a time and place for eventual consistency and multi-master, and there is a time for strongly consistent SQL servers.
Both have their place, but I think you get better architected, more reliable software with an eventually consistent system.
In and eventually consistent system, the programmer must consider what to do in the case of data failure. I think this is a good thing. When building a user authentication system, who better to decide what to do when you can't access the user database than the person writing the user auth system? Should it fail open or closed? Well if you're a bank, you should probably fail closed. If you're Netflix? Fail open. The programmer should know the business case better than the database, which will alway fail closed.
Sure, in a consistent system the programmer could think about that, but they aren't forced to, which I think is good.
Also, it means that joins must be done in your application. Sometimes its good to have the programmer deal with that too, because it means they will have to really consider what data they need and be much more familiar with how much data is needed to create the join. If you're just relying on the database to do the joins, it might be hiding a lot of load from you, especially if you inadvertently create multiple table scans. In an eventually consistent system, you know if you created a table scan, because you have to pull back a full table's worth of data (or not, because you saw that issue and fixed it).
I just feels like the programmer gets better insight into what is happening to the data and can make better decisions on what to do when they are closer to the business cases the software is being built for.