News for hackers. Stuff that matters.
Re:A Conflict-Free Replicated JSON Datatype by espadrine on Wednesday August 17, 2016 @04:47AM (#12303223)
> Our principle of not losing input due to concurrent modifications appears reasonable, but as illustrated in Figure 4, it leads to merged document states that may be surprising to application programmers who are more familiar with sequential programs.

That decision is a bit odd to me. Not only do they show that their merge system can produce data that the application considers invalid, it can even convert a string to a list without having any operation creating a list explicitly.

> Moreover, garbage collection (tombstone removal) is required in order to prevent unbounded growth of the datastructure

That is always the painful part of a CRDT system. Until that part is done, it cannot really be used in production. That said, previous work on CRDTs give me confidence that it can be implemented with minimal overhead.

> In fact, we wrote the LATEX source text of this paper using an experimental collaborative text editor that is based on our implementation of this CRDT. We are making the character-by-character editing trace of this document available as supplemental data of this paper

String editing is brushed over in the paper. I'd love to see their implementation. Does anyone know where it all is?

Re:A Conflict-Free Replicated JSON Datatype by nateps on Wednesday August 17, 2016 @01:16PM (#12306368)
We've been developing ShareDB (https://github.com/share/sharedb), a JSON OT system, and using it in production at Lever for 3 years now.

CRDT has some strong advantages in decentralized applications with a need to avoid a central server. But in practice, web apps today do generally have a central server.

I think the core advantages of OT over CRDTs are actually these practical concerns: being able to design transformations that are more intuitive to application developers, easily cleaning up old operation history, and real world production use and examples of the technology already.

One thing OT has enabled us to do that you cannot do as easily in a CRDT is to support turning off publishing of operations when we do large migrations and create huge numbers of ops at once. Because ops have a strict ordering and versioning, clients can lazy get the ops that they have missed if needed later because they submitted a modification on a document or see an even newer op and need the intermediate ops. In a CRDT, you have to make sure that all the clients get all the ops in order to converge. It sounds simple but can be complex in practice.

Another practical issue I'd add is that these algorithms sometimes place the burden of resolving the outcome of concurrent edits at read time vs. write time. With OT, you generally do a lot of complex work to figure out how to transform ops at write time, but when you read, you just read the already fully updated document in its native format. CRDT systems often store the document in a format that can be written to very cheaply as it is commutative, but the work gets pushed to read time when you have to collapse a tree full of all its history into a different data structure, such as a string for text editing. One is not strictly better than the other, but many production systems are vastly more read heavy than write heavy, so less obvious tradeoffs like this can become extremely important in production use.

Re:A Conflict-Free Replicated JSON Datatype by arj on Wednesday August 17, 2016 @03:11PM (#12307468)
That last part is a good summary of CRDT versus OT.

Also nice project. I think a section on conflict handling would be a good addition to your README on github. It is the first thing I look for when looking at systems in this space :)

Re:A Conflict-Free Replicated JSON Datatype by tracnar on Monday August 22, 2016 @09:27AM (#12336142)
Have you read the SwiftCloud paper? https://pages.lip6.fr/Marek.Zawirski/papers/SwiftCloud-RR-83...

It's about using a centralized cloud infrastructure in conjunction with CRDTs. This allows to fix some of the issues you mention (pruning of operation history, stricter versioning, no need to push operations, ...).

Of course it complicates the implementation but it's quite interesting.

Re:A Conflict-Free Replicated JSON Datatype by jpallen on Wednesday August 17, 2016 @06:04AM (#12303483)
>> In fact, we wrote the LATEX source text of this paper using an experimental collaborative text editor that is based on our implementation of this CRDT. We are making the character-by-character editing trace of this document available as supplemental data of this paper

If anyone else wants to feel at the cutting edge of science, you can write your LaTeX paper collaboratively at https://www.sharelatex.com :P

I can't pretend we've got any original algorithms (we lean on ShareJS a lot, which is operational transform based), but we've got a few years and billions of key strokes of experience in making it robust and practical.

Re:A Conflict-Free Replicated JSON Datatype by eternalban on Wednesday August 17, 2016 @02:05PM (#12306796)
> [Garbage collection] is always the painful part of a CRDT system.

This is not unique to CRDTs: a temporal datastructure will accumulate garbage. Blockchain (e.g. Git); MVCC (e.g. Postgresql).

Re:A Conflict-Free Replicated JSON Datatype by ergl on Wednesday August 17, 2016 @05:57AM (#12303467)
About garbage collection, there is some research done in that direction: Logoot [0] is a collaborative text editor that doesn't depend on tombstones.

[0]: https://hal.archives-ouvertes.fr/inria-00336191/

Re:A Conflict-Free Replicated JSON Datatype by nkohari on Wednesday August 17, 2016 @08:56AM (#12304255)
I've been doing some tinkering in this area lately, and I was also looking for an explanation of the string editing mechanics in this algorithm.

The hardest part there is retaining the intent of the simultaneous edits -- the I in the "PCI Consistency model" as described by the WOOT paper. Previous algorithms (LSEQ, WOOT, Doctree, Logoot, etc.) devoted to simultaneous conflict-free editing of text have solved this through relatively complex key generation strategies. I've only done a cursory read of this paper, but this algorithm doesn't seem to implement an alternative solution.

Re:A Conflict-Free Replicated JSON Datatype by vidarh on Wednesday August 17, 2016 @12:56PM (#12306169)
It's not trying to do string editing at all.

It's operating on JSON de-serialized to maps and lists. String editing isn't a valid operation on the data model they specify. You can replace a string value with another.

Section 3.3 outlines the allowed operations, which basically allows you to obtain an iterator, move it, insert, delete and retrieve keys and values.

You could certainly model string editing in it, by putting each character into it as a separate value, but then as you point out it'd probably not do very well because it doesn't contain heuristics to model intent of text editing, but to model changes to a JSON object hierarchy while avoiding loss of data.

You could use this algorithm combined with one intended for string-editing to improve on it for values where partial edits of the values matters, certainly.

Re:A Conflict-Free Replicated JSON Datatype by nicolagreco on Sunday August 21, 2016 @05:50AM (#12330199)
wouldn't a string just be a list of chars?
Re:A Conflict-Free Replicated JSON Datatype by vidarh on Wednesday August 17, 2016 @05:03AM (#12303278)
> That decision is a bit odd to me. Not only do they show that their merge system can produce data that the application considers invalid, it can even convert a string to a list without having any operation creating a list explicitly.

It's hard to see good alternatives there. If you move from a single editor to concurrent edits without very strict ordering guarantees or a transactional system, you will need to deal with conflicts or lose data. Creating output that would be considered invalid unless the application is fixed to explicitly deal with it (or explicitly throw it away) seems quite reasonable.

Re:A Conflict-Free Replicated JSON Datatype by adrianratnapala on Wednesday August 17, 2016 @05:31AM (#12303386)
For this you would want a type / schema that explicitly represented the conflicts at the right level of granularity and helped applications resolve them.

Whereas producing an object the pretends to be conflict free, but has some fields that might or might not change type is the worst of both worlds.

Re:A Conflict-Free Replicated JSON Datatype by disordinary on Wednesday August 17, 2016 @09:33AM (#12304525)
Exactly, if there are conflicts that have to be resolved then it's not conflict-free.
Re:A Conflict-Free Replicated JSON Datatype by vidarh on Wednesday August 17, 2016 @12:48PM (#12306070)
It is conflict free in this sense if no change is rejected, and if the result eventually converges (e.g. if two parties receive the full set of changes, they will end up with the same result, but they may diverge locally until all changes have been synchronised to all replicas).

This does mean sometimes producing something that have "application level conflicts" or something that a user will consider a conflict even if the application is perfectly fine with it. Depending on how good the heuristics for determining which changes to keep are, this may be obnoxious or appear perfectly reasonable.

E.g. two users start with the same empty document. They each type a sentence. Merge happens. Did they intend the document to have just their sentence, or do they want both? User A wrote that their mutual boss is an idiot; user B wrote that their mutual boss is a genius. Now what do they want to remain? There's no way for software to resolve that automatically which doesn't have disturbing implications.

But there certainly are some alternatives that are better than others. E.g. you'd probably prefer to retain both sentences, so you can agree on which to keep. You'd certainly prefer to keep at least one to deleting both.

Re:A Conflict-Free Replicated JSON Datatype by vidarh on Wednesday August 17, 2016 @12:14PM (#12305761)
They are presenting an algorithm, not a product, so any implementation would be free to make it available in a different way.

They specifically do address this point about how developers would deal with it in the conclusion of the paper.

Re:A Conflict-Free Replicated JSON Datatype by amelius on Wednesday August 17, 2016 @05:20AM (#12303346)
> String editing is brushed over in the paper.

LaTeX source code is basically just a string. It would have been more interesting to see this applied to a hierarchical data-structure, such as HTML, where e.g. elements are nested inside markup nodes.

Re:A Conflict-Free Replicated JSON Datatype by Kristine1975 on Wednesday August 17, 2016 @08:02AM (#12304001)
LaTeX is also hierarchical, though. Some is explicit (environments) and some is implicit (\chapter, \section etc).
Re:A Conflict-Free Replicated JSON Datatype by nateps on Wednesday August 17, 2016 @12:38PM (#12305957)
This is true. To preserve document validity and user intent, an algorithm designed for collaborative string editing will not always work correctly when applied to a markup document such as LaTeX, HTML, or Markdown directly.

It is similar to how you need a JSON type to represent the kinds of edits you can do to a JSON document rather than editing JSON text as a string. The kinds of edits done in text better map to editing a markup language like LaTeX or Markdown than JSON, so you are less likely to notice these issues right away. But if the type is designed specifically for the markup language, user intent will be better maintained through concurrent edits and the document will never end up in an invalid format.

Re:A Conflict-Free Replicated JSON Datatype by amelius on Wednesday August 17, 2016 @12:50PM (#12306087)
The same holds for LaTeX stored in git, yet a lot of people use this combination without problems.

It is a hack, in a sense, but one without serious consequences.

Re:A Conflict-Free Replicated JSON Datatype by nateps on Wednesday August 17, 2016 @01:32PM (#12306500)
Agreed. It is likely to work pretty well with a format that is very close to plain text like LaTeX most of the time.

Git's resolution strategy is diff-match-patch of text, so it is a good analogy. However, the thing that Git does that CRDTs and OT string types generally do not is create conflicts requiring user intervention before proceeding.

If you are realtime editing concurrently, this might be acceptable, because the users might be able to see the conflict and resolve it. However, I'd say a clear UI for conflicts is the key reason why Git is able to more intuitively and safely deal with concurrent edits of non-plain text using a plain text algorithm.

Re:A Conflict-Free Replicated JSON Datatype by amelius on Wednesday August 17, 2016 @08:25AM (#12304114)
You already indicate that the LaTeX model is not a "clean" hierarchy. So I highly doubt they modeled it like that in JSON, but it could be possible.
Re:A Conflict-Free Replicated JSON Datatype by wheresmypwd on Wednesday August 17, 2016 @02:32PM (#12307109)
It's a hierarchy like html, where p tags have implicit close tags.
Re:A Conflict-Free Replicated JSON Datatype by tveita on Wednesday August 17, 2016 @11:35AM (#12305458)
Reminds me of the operational transformation used in Wave.

http://www.codecommit.com/blog/java/understanding-and-applyi...

I see they mention it in the paper, saying the difference is that Wave requires a central server to impose a total ordering of operations.