News for hackers. Stuff that matters.
Re:Crsql – Multi-writer and CRDT support for SQLite by maxpert on Tuesday November 15, 2022 @09:33AM (#33609580)
Ideas in CRSQL are remarkably similar to Marmot (https://github.com/maxpert/marmot) except that it doesn't require an extension to be installed, and it can run as sidecar. I can see how having extension makes it simple to intercept writes, and transactions. Since marmot builds on top of NATS, and JetStreams work using Raft, it can easily scale and keep the libsqlite on disk guarantees intact.
Re:Crsql – Multi-writer and CRDT support for SQLite by lifty on Tuesday November 15, 2022 @10:40AM (#33610487)
Marmot looks interesting, but I am curious how it deals with conflicts. What happens if two writes for the same field happen on two different dbs, and they are conflicting?
Re:Crsql – Multi-writer and CRDT support for SQLite by maxpert on Tuesday November 15, 2022 @02:05PM (#33613451)
Since each write is logged into a JetStream. The last writer will win for conflicting changes.
Re:Crsql – Multi-writer and CRDT support for SQLite by lifty on Tuesday November 15, 2022 @04:13PM (#33615093)
But since you don't have NATS message ordering (from different sources), it's possible that different DBs in the cluster will end up out of sync, right?
Re:Crsql – Multi-writer and CRDT support for SQLite by maxpert on Tuesday November 15, 2022 @06:46PM (#33616879)
So per-node there is an order of changes (read internals doc), and all of changes are sent to JS in that commit order. These changes when replayed will always eventually result in same values.
Re:Crsql – Multi-writer and CRDT support for SQLite by lifty on Wednesday November 16, 2022 @04:48AM (#33620790)
I am talking about writes that come from 2 different nodes. If 2 different nodes, each making a different write at the exact same time, you would need some global ordering or consensus in order to order them. Otherwise you might end up in a different state on different nodes.
Re:Crsql – Multi-writer and CRDT support for SQLite by tantaman on Tuesday November 15, 2022 @11:52AM (#33611655)
I'd originally done a sidecar route but ultimately went the extension route for portability.
Re:Crsql – Multi-writer and CRDT support for SQLite by maxpert on Tuesday November 15, 2022 @06:51PM (#33616941)
It’s an interesting choice for sure because I heard a lot of ifs and buts for loads of eventually consistent scenarios on why people wanted to not make any changes to their code and don’t really wanna load extensions. Loading extensions is definitely better in terms of transaction control. I did a lot of soul-searching before settling for sidecar approach. In order to not be an interface and stay away from binding complications for various platforms I opted for sidecar. Once done I was already able to scale out of box stuff like PocketBase.