News for hackers. Stuff that matters.
Re:The Session Extension by api on Monday December 13, 2021 @03:10PM (#29543966)
SQLite plus a fast Raft replication and consensus implementation is the next Postgres. Change my mind.
Re:The Session Extension by rocmcd on Monday December 13, 2021 @03:43PM (#29544370)
This exists already with rqlite: https://github.com/rqlite/rqlite

As much as I love the idea, Raft is not a silver bullet and can break in strange and inconspicuous ways that may be difficult to fix. It can also lead to subtle inconsistencies in the data depending on what data you are putting in it. Overall it may be good in some situations, but I doubt it will dislodge Postgres in any meaningful way.

Re:The Session Extension by otoolep on Monday December 13, 2021 @05:38PM (#29545605)
What inconsistencies do you mean, precisely? I think you need to provide a clear example of what you mean, to understand what you're getting at.

In my experience, any inconsistencies would be due to misuse of Raft, not something intrinsic to Raft itself.

Re:The Session Extension by rocmcd on Tuesday December 14, 2021 @08:11AM (#29551117)
Thanks for chiming in, Philip! I’m a big fan of your work.

I was referring to the issues around determinism in queries. These are easy enough to catch if you’re aware of them, but it pushes more onus onto the application layer / devs (not that it’s necessarily a bad thing, just a trade-off).

Re:The Session Extension by Conlectus on Monday December 13, 2021 @04:10PM (#29544680)
Wouldn't this be true of any form of replication, including whatever Postgres uses? Or is the proof of correctness for Raft weaker than some other class of consensus?

Edit: from rqlite's FAQ

> Raft is a Consistency-Partition (CP) protocol. This means that if a rqlite cluster is partitioned, only the side of the cluster that contains a majority of the nodes will be available.

Disconcertingly, this sounds like they assume that only one side of a partition can contain a majority of nodes. This isn't true if the partition is partial, eg A<-->B, B<-->C, A<-/->C.

Re:The Session Extension by otoolep on Monday December 13, 2021 @05:36PM (#29545594)
rqlite creator here.

rqlite (and Raft) doesn't assume this. All it states is that for the cluster to make progress i.e. apply a change in a consistent manner, at least a quorum of nodes must be online and in contact with the Leader (which is one of the online nodes). Every node in a rqlite cluster knows what size the cluster is and therefore requires that (N/2)+1 nodes acknowledge the change before that change is committed. The Leader node performs this coordination.

The scenario you outlined above is obviously possible. But in that event the cluster is down -- the Leader (let's say it's node A) cannot contact a quorum of nodes. No changes can be made to it.

Re:The Session Extension by Conlectus on Tuesday December 14, 2021 @11:51AM (#29553824)
Gotcha, that makes sense. I look forward to digging into the code myself :).
Re:The Session Extension by pstuart on Monday December 13, 2021 @08:08PM (#29546915)
rqlite is nice but it's got limitations that made me look elsewhere. https://dqlite.io/ does it in C and allows for transactions and can be used for the Go sql.DB object.
Re:The Session Extension by howdydoo on Monday December 13, 2021 @03:22PM (#29544121)
https://www.sqlite.org/whentouse.html

See: Situations Where A Client/Server RDBMS May Work Better

Re:The Session Extension by thegeekpirate on Monday December 13, 2021 @03:39PM (#29544331)
Re:The Session Extension by tptacek on Monday December 13, 2021 @05:13PM (#29545356)
Multi-writer concurrency with sqlite is a giant pain in the ass.
Re:The Session Extension by mst on Monday December 13, 2021 @11:47PM (#29548345)
There are, nonetheless, lots of situations where I would have reached for pg automatically a few years ago where I now stop and think about which of the two is going to fit my use case better.

The comment you're replying to is saying rather more than that, and I don't endorse it, but the shift in the possibility space is still both fascinating and impressive.

(also I love sqlite's tendency to use pg as a syntax template, it seems like an example of two projects with significantly different goals playing to each others' strengths in a beautifully positive sum game fashion)

Re:The Session Extension by pstuart on Monday December 13, 2021 @08:10PM (#29546938)
The writers need to be serialized but it's not that big a pain.
Re:The Session Extension by le-mark on Monday December 13, 2021 @07:04PM (#29546426)
Multiple reader single writer out performs Postgres in my testing, by a large margin.
Re:The Session Extension by penberg on Thursday December 16, 2021 @02:19AM (#29575890)
I don't know if SQLite + Raft is going to be the next Postgres, but it's certainly an interesting solution for a variety of use cases where you don't need your data _partitioned_ on multiple machines (but still want it to be replicated for availability and fault tolerance).

As others point out in this thread, there's already rqlite for Go, sqlite for C, and now we've open sourced our work on bringing SQLite and Raft to Rust: https://glaubercosta-11125.medium.com/winds-of-change-in-web...

Re:The Session Extension by remram on Monday December 13, 2021 @06:38PM (#29546211)
Adding Raft and replication will make your database more available but strictly slower and not any more concurrent. If SQLite is not good enough for you now it probably wouldn't be any good for you then.
Re:The Session Extension by lrem on Monday December 13, 2021 @05:57PM (#29545794)
SQLite is not a high throughput database. Trying to work around this with cool tech is not likely to make it better than a boring solution, like using a high throughput database in the first place.
Re:The Session Extension by bob1029 on Monday December 13, 2021 @06:30PM (#29546143)
My SQLite usages are able to saturate my NVMe storage devices in many test cases. Not sure what exact criteria you are using for "high throughput" here. Perhaps you are not familiar with the additional WAL journal mode or synchronization flags?
Re:The Session Extension by cryptonector on Monday December 13, 2021 @08:25PM (#29547022)
SQLite3 is not a high-write-concurrency database, but not a high-throughput DB is less clear, and it's not at all clear that it can't be hacked on to go faster.

If you take a sharding approach to concurrency, you might just be happy with no write concurrency: one process/thread per-CPU / storage slice, and fix everything else upstairs. It's not a crazy approach, and it might be the approach many would take if they could start from scratch. In fact, it's the approach Spanner takes, for example.