News for hackers. Stuff that matters.
Re:Every System is a Log: Avoiding coordination in distribut... by pjc50 on Friday January 24, 2025 @10:39AM (#42814134)
> Having a single place (the one log) that forces a linear history of events as the ground truth and owns the decision of who can add to that ground truth, means we don’t have to coordinate much any more.

Well, yes, but then you've backed into CAP again because you only have one log.

Re:Every System is a Log: Avoiding coordination in distribut... by sewen on Friday January 24, 2025 @11:00AM (#42814383)
Yes, we are assuming a log that picks linearizability at the cost of availability under partitions. Like most logs do, including Kafka, Pulsar, RedPanda, etc.

The application state is defined by the log here, and the log drives retries/recovery, so it doesn't much matter if the process that executes the app code splits off. The log would hydrate another one.

Also the one log is at the granularity of a single key or handler execution. More of a logical log, than a physical log or even partition.

In Restate, we implement a logical log-per-key, backed by a partitioned physical log.

Re:Every System is a Log: Avoiding coordination in distribut... by UltraSane on Friday January 24, 2025 @12:02PM (#42815090)
couldn't the log be synchronously replicated to multiple servers to increase A?
Re:Every System is a Log: Avoiding coordination in distribut... by ismailmaj on Friday January 24, 2025 @01:26PM (#42815793)
In case of mutation either the replicas will be out of sync (so no C), or you'll need to synchronously mutate the replicas (so bad A due to latency).
Re:Every System is a Log: Avoiding coordination in distribut... by UltraSane on Friday January 24, 2025 @01:49PM (#42815964)
delaying writes by a factor longer than the max clock skew of the cluster is a pretty common strategy. It is what Google Spanner does.
Re:Every System is a Log: Avoiding coordination in distribut... by ismailmaj on Friday January 24, 2025 @02:05PM (#42816077)
Right but it's not magic as you cannot predict networking delays.

The max clock skew also has to pick between A and C, in underestimates you lose serializability, in overestimates you pay in write latency.

Re:Every System is a Log: Avoiding coordination in distribut... by UltraSane on Friday January 24, 2025 @02:50PM (#42816374)
What if you use CockroachDB for your log? They do something pretty clever: https://www.cockroachlabs.com/blog/living-without-atomic-clo...
Re:Every System is a Log: Avoiding coordination in distribut... by ismailmaj on Saturday January 25, 2025 @07:57AM (#42821367)
Thanks for the link, not much familiar with CockroachDB but I worked on a similar system that wanted to provide strong consistency when reading from many databases, some of which where Spanner but not all of them.

Eventually the system implemented something similar to "find transaction timestamp at execution time" + optimistic locking, as the article states you lose global linearizability and have to do some re-reads for writes.

What I missed the most from Spanner compared to the system above is that its tricky to do client-side transaction chaining where there are many clients and the order is important.

Re:Every System is a Log: Avoiding coordination in distribut... by UltraSane on Saturday January 25, 2025 @09:28AM (#42821842)
If you put a atomic clock like OCP-TAP Time Card with GNSS receiver in every server and use precision time protocol to sync them you should be able to synchronize quite a few servers to less than 100 nanoseconds.
Re:Every System is a Log: Avoiding coordination in distribut... by mrkeen on Friday January 24, 2025 @10:45AM (#42814213)
If I've understood it, it's like using Kafka with 1 topic and 1 partition. But it shouldn't rule out multiple brokers with a >1 replication factor, giving you CP.
Re:Every System is a Log: Avoiding coordination in distribut... by clayg on Friday January 24, 2025 @10:43AM (#42814193)
But can't any log be implemented as a CRDT? Was that not implied in the post? I didn't read it that close...
Re:Every System is a Log: Avoiding coordination in distribut... by ismailmaj on Friday January 24, 2025 @10:50AM (#42814273)
CRDT is really only useful when inconsistencies could be acceptable in some situations and so it depends on the application.

For something that is trying to solve the general problem of consistent single ground-truth log, you can't really do much better than Spanner.

Re:Every System is a Log: Avoiding coordination in distribut... by logsr on Friday January 24, 2025 @12:34PM (#42815419)
the "conflict-free" in crdt is like miller high life being the "champagne of beers." it really means conflicts ignored and conflicting items discarded by established rules, which works well for some use cases, but for many does not.