News for hackers. Stuff that matters.
Re:Is Redlock Safe? Reply to Redlock Analysis by jedberg on Tuesday February 09, 2016 @11:16AM (#11066258)
If you need a global lock in your distributed system, then you're already in trouble. In most cases, the best solution is to rewrite your software so that you don't need the lock. I don't want to say all, because I'm sure there are use cases where it would be impossible to do so. However, for my own research, I'm having trouble finding examples where a global lock is absolutely necessary. If anyone has any examples, please send them to me.

For example, a lot of people say, "updating a bank balance". But even that can easily be eventually consistent. You send increments and decrements and settle up at the end. Yes, your balance might go negative, but that's what ATM limits are for -- the most you could go negative is that limit. Most other systems can be written the same way. And for non-numeric values, there are always vector clocks or timestamps or quorum to determine which update came next, or you could send a client a list of all recent updates and let the client "do the right thing".

Re:Is Redlock Safe? Reply to Redlock Analysis by antirez on Tuesday February 09, 2016 @11:20AM (#11066307)
While I agree with the general sentiment, the matter is, there were a ton of people using a single Redis instance for distributed locks, since people find this model useful sometimes, even when best alternatives exist, or maybe because of limitations of systems they interact with. So I tried to provide something better compared to what people were using, but comparatively easy.
Re:Is Redlock Safe? Reply to Redlock Analysis by jedberg on Tuesday February 09, 2016 @11:28AM (#11066399)
I think what you have provided is a far better solution than how most people were using redis. :) I was just hoping to convince people that they should think a little deeper before going straight for the global lock.

Hope you're doing well BTW!

Re:Is Redlock Safe? Reply to Redlock Analysis by biznickman on Tuesday February 09, 2016 @03:13PM (#11068389)
Have definitely thought through this process and wound up using Redlock for the time being.

I didn't expect it to be a perfect system but I don't want to deal with setting up a queuing system right now when there simply isn't enough activity taking place.

Will it fail at some point? Possibly, but it's a great intermediary solution as you state. I for one would be curious to see other solutions to this problem. Are there any resources you'd recommend?

Re:Is Redlock Safe? Reply to Redlock Analysis by pron on Tuesday February 09, 2016 @11:38AM (#11066500)
That may be the case, but all other protocols that guarantee consistency suffer from similar subtle issues (which can be resolved through careful algorithm design). Eventual consistency, however, is never the answer, unless your question is "how do I get eventual consistency?" The consistency required by an application is entirely imposed by the domain, and it is that requirement that causes algorithmic difficulties. Using less-consistent protocols to achieve a consistent view simply pushes the problem to a different layer.

Also, eventual consistency is a weak requirement that, when sufficient, enables a certain level of fault tolerance and low latency not achievable with other consistency levels (see [1]), although other consistency levels may offer good-enough fault-tolerance and latency (rivaling EC in many cases). EC is not in general the solution to the problem of "how do I best distribute my data?"

[1]: http://www.bailis.org/blog/when-does-consistency-require-coo...

Re:Is Redlock Safe? Reply to Redlock Analysis by jedberg on Tuesday February 09, 2016 @11:42AM (#11066551)
> Using less-consistent protocols to achieve a consistent view simply pushes the problem to a different layer.

Absolutely, but if you can avoid using global locks, which are error prone and reduce reliability, isn't that a good thing?

Re:Is Redlock Safe? Reply to Redlock Analysis by pron on Tuesday February 09, 2016 @12:54PM (#11067188)
It's not a question of good or bad. Providing a higher level of consistency than what your application needs will only hurt you, but many applications just cannot do with weak consistency.
Re:Is Redlock Safe? Reply to Redlock Analysis by derefr on Tuesday February 09, 2016 @06:55PM (#11069797)
I think the point being made here is that "what your application needs" is often something you chose your way into, and can choose your way back out of by backing out some other constraints—and that it's often more sensible to back out these constraints than to dive head-first into a problem domain that requires distributed locking.

In other words: if you set out to create Bitcoin in 1998, you might end up running into distributed consensus problems that are difficult enough that you decide to "solve an easier problem"—such that you end up creating Paypal instead.

Re:Is Redlock Safe? Reply to Redlock Analysis by pron on Tuesday February 09, 2016 @11:01PM (#11070681)
> I think the point being made here is that "what your application needs" is often something you chose your way into, and can choose your way back out of

I think that while this may be true for many requirements, consistency is among the least flexible ones. Besides, losing consistency usually makes your system more complicated, not less. Weak consistency has some real benefits (latency in some cases, tolerance for more failure modes), but simplicity is not usually one of them.

> In other words: if you set out to create Bitcoin in 1998, you might end up running into distributed consensus problems that are difficult enough that you decide to "solve an easier problem"—such that you end up creating Paypal instead.

I don't think that the design problems behind Bitcoin are any easier than distributed consensus, which, BTW, is not necessary for quite a few consistency levels.

Re:Is Redlock Safe? Reply to Redlock Analysis by YZF on Tuesday February 09, 2016 @11:20AM (#11066313)
What happens when you sent the increment and then failed to send the decrement? The problem isn't typically the eventual consistency aspect, though that can also be a problem since you want the balance to be accurate, but rather the transactional nature.
Re:Is Redlock Safe? Reply to Redlock Analysis by jedberg on Tuesday February 09, 2016 @11:29AM (#11066409)
How would a global lock fix this? Presumably you would have a process that is trying to decrement get a global lock and then crash, and eventually release the lock anyway, right? So you get the same outcome.
Re:Is Redlock Safe? Reply to Redlock Analysis by YZF on Tuesday February 09, 2016 @11:56AM (#11066681)
You can build things like 2 phase locking on top of that. You're right though that simply taking a lock before updating two values doesn't solve the atomicity of that update problem. However if you're updating a single value then taking a lock can make your life easier vs. systems with no locks and eventual consistency.
Re:Is Redlock Safe? Reply to Redlock Analysis by jedberg on Tuesday February 09, 2016 @12:03PM (#11066749)
You can make your life as a programmer easier in exchange for making your life as an operator harder. :) Locks are brittle and hurt reliability.
Re:Is Redlock Safe? Reply to Redlock Analysis by newman314 on Tuesday February 09, 2016 @11:26AM (#11066372)
I'd imagine it being pretty unsettling to watch a bank balance fluctuate. Also, what if there are downstream operations that depend on the value being accurate I.e. Trigger overdraft condition?
Re:Is Redlock Safe? Reply to Redlock Analysis by jedberg on Tuesday February 09, 2016 @11:34AM (#11066460)
> I'd imagine it being pretty unsettling to watch a bank balance fluctuate.

Banks have generally solved this by batching updates, but yes, it could be unsettling for the customer in a similar situation. There are ways around that though, for example you update a cache that stores the data instantly and then let the eventually consistent source of truth update the cache again if it is not in sync, or perhaps have a short timeout on the cache, so the customer gets a consistent and smooth experience but the source of truth is still accurate, eventually.

Re:Is Redlock Safe? Reply to Redlock Analysis by eva1984 on Tuesday February 09, 2016 @11:29AM (#11066407)
Is leader election somehow a usage of a global lock? I do agree it should be avoided actively though.
Re:Is Redlock Safe? Reply to Redlock Analysis by jodah on Tuesday February 09, 2016 @12:11PM (#11066828)
It's related.

Leader election generally[1] requires consensus among distributed processes, and global locks generally[1] require consensus as well. The benefit of this is that both problems can be solved on top of a common consensus implementation which is what http://atomix.io does.

1: I say generally because you can do fancy things with fancy clocks to avoid running operations through a quorum under certain circumstances, but these carry caveats that preclude them from being reliable enough to use in many use cases.

Re:Is Redlock Safe? Reply to Redlock Analysis by jedberg on Tuesday February 09, 2016 @11:30AM (#11066430)
I don't think so, there are distributed leader election schemes that don't require a lock. The simplest is using HRW hashing[0] so that each node independently determines the same leader.

[0] https://en.wikipedia.org/wiki/Rendezvous_hashing