News for hackers. Stuff that matters.
Re:Resilient Sync for Local First by threatofrain on Tuesday June 25, 2024 @02:29AM (#40785144)
CRDT's largely can't rectify merge conflicts... what CRDT's can do is make your merges convergent. If I auto-merged your Git code using a blunt policy I don't know if you'd reach for "correct" as your first description as opposed to "convergent".

The longer you're offline the more you have to explain to your users the circumstances where they might need to do manual data cleanup or expect creative merging policy. No matter what tech you're using.

Re:Resilient Sync for Local First by ErikBjare on Tuesday June 25, 2024 @03:19AM (#40785465)
This is true if you're working with documents edited by different devices, but many CRDTs do not and can indeed be not only "convergent" but also "correct".

How SSB works is a good example.

Re:Resilient Sync for Local First by vlovich123 on Tuesday June 25, 2024 @09:22AM (#40788296)
Can’t find what SSB is. Can you clarify?

But yes, it’s possible there are applications that can constrain the set of operations such that the merged result is also correct. I was just highlighting that CRDTs are trotted out as a catch all when a) it’s just a way of thinking about the problem rather than an actual algorithm (eg while there’s a lot of CRDT work for document editing, that work has to be done for scratch for any other problem domain) b) you have to modify your problem constrains and solution such that merge results are correct. B is a very very hard problem in a distributed system even if you ignore the challenge of a which itself is quite hard. I wouldn’t trust anything in this space too much without a TLA+ proof that it’s correct (unless it’s something low stakes like document editing).

Re:Resilient Sync for Local First by endgame on Tuesday June 25, 2024 @12:19PM (#40790383)
I think SSB is "Secure Scuttlebutt": http://ssbc.github.io/ssb-db/
Re:Resilient Sync for Local First by threatofrain on Tuesday June 25, 2024 @04:13PM (#40793056)
Some of our most important DB sync'ing algorithms were only very recently verified. It was okay to run the whole world's systems, including hospitals and armed forces this way. Very very few industry algos receive formal verification.
Re:Resilient Sync for Local First by kiitos on Tuesday June 25, 2024 @04:50AM (#40786070)
CRDTs solve merge conflicts by construction, because they necessarily don't permit merge conflicts in the first place.
Re:Resilient Sync for Local First by xnorswap on Tuesday June 25, 2024 @05:56AM (#40786490)
By definition they don't have conflicts in the technical sense of the word, but they can easily get into bad states after a merge.

It's not technically a conflict since there is a path forward, but the resulting output can be nonsensical.

For example you could employ a strategy where given two concurrent edits, the merge "randomly*" picks one of the two edits for each property/fact, and abandons the other.

That's a construction of a conflict free replicated data type, but it says nothing of the quality of the final result.

Ultimately, the quality comes down to how well you can produce merge strategies for your domain.

* "Randomly" in a deterministic way, e.g. by comparing a hashes of transaction IDs or similar.

Re:Resilient Sync for Local First by kiitos on Tuesday June 25, 2024 @04:01PM (#40792930)
> By definition they don't have conflicts in the technical sense of the word...

...which is the sense that we're talking about, when we talk about CRDTs.

Re:Resilient Sync for Local First by Dylan16807 on Tuesday June 25, 2024 @06:43PM (#40794547)
When someone goes into detail about correct versus convergent, they are explaining how the technical sense is not what people really want, and you bluntly replying that it's solved (because it's technically solved) is unhelpful at best and misleading at worse.
Re:Resilient Sync for Local First by kiitos on Thursday June 27, 2024 @12:25AM (#40807202)
A CRDT is a purely technical construct! Correctness (in terms of merge conflicts) is well-defined in that context and it's orthogonal to any kind of semantic/application definition!

This is a bonkers conversation. It's like claiming integer division isn't correct because 5/2 should be 2.5 instead of 2.

Re:Resilient Sync for Local First by bombela on Tuesday June 25, 2024 @05:13AM (#40786218)
Alright, try this in Google Doc.

Two users simultaneously select the same word, and replace it by a different word each.

You might end up with: "apple" -> "carrot orange"

Sure it technically merged, and did not lose data. But you still need humans to reconcile the final state anyways. Worse, you might not even notice that you need to cleanup.

Re:Resilient Sync for Local First by vlovich123 on Tuesday June 25, 2024 @09:24AM (#40788319)
Bad example. Google docs doesn’t use CRDTs but uses OT instead. CRDTs may handle your scenario just fine depending on how they decide to handle this scenario.

The easier scenario that is independent of the specific algorithm is the one I described (have follow up comments describing it in more explicit detail) where you have a logical meta instruction within the document and then the document is updated based on outdated meta instructions offline but when they come back online the meta instructions were changed.

Then there’s not even a merge conflict to really worry about.

Re:Resilient Sync for Local First by mattarm on Tuesday June 25, 2024 @11:32AM (#40789879)
> Bad example. Google docs doesn’t use CRDTs but uses OT instead. CRDTs may handle your scenario just fine depending on how they decide to handle this scenario.

The CRDT may pick one or the other replacement word, but who is to say that either choice is correct? Perhaps including both words is correct.

> Then there’s not even a merge conflict...

Agree, this is what CRDTs are all about.

> ...to really worry about.

I think it is important to make clear that CRDTs do not "solve" the merging problem, they merely make it possible to solve in a deterministic way across replicas.

Often, CRDTs do not capture higher level schema invariants, and so a "conflict free" CRDT merge can produce an invalid state for a particular application.

There is also the example above, where at the application level, one particular merge outcome may be preferred over another.

So, it isn't as simple as having nothing to worry about. When using CRDTs, often, there are some pretty subtle things that must be worried about. :-)

Re:Resilient Sync for Local First by vlovich123 on Tuesday June 25, 2024 @02:25PM (#40791807)
Yup 100% agreed.
Re:Resilient Sync for Local First by kiitos on Thursday June 27, 2024 @02:10AM (#40807777)
> The CRDT may pick one or the other replacement word, but who is to say that either choice is correct?

You, as the application developer.

> I think it is important to make clear that CRDTs do not "solve" the merging problem

They literally do, in the context in which they are defined. Which is about data consistency, not semantic correctness.

> Often, CRDTs do not capture higher level schema invariants,

CRDTs never capture higher level schema invariants. Just like TCP doesn't enforce HTTP session authentication. Orthogonal concerns.