Embeddable, replicated and fault tolerant SQL engine based on Sqlite

Re:Embeddable, replicated and fault tolerant SQL engine base... by Labo333 on Tuesday February 01, 2022 @06:42PM (#30171412)

What I found to be the biggest problem with SQLite is concurrent write operations. WAL helps for concurrent read but not for concurrent write.

On a moderate use case (< 20 write operations per s) I kept getting timeouts so I first devised a quick fix system using RQ jobs passing writes to a single process and migrated to postgres as fast as I could.

Does this software help with concurrent write operations?

[ Parent | on HN ]

Re:Embeddable, replicated and fault tolerant SQL engine base... by bob1029 on Tuesday February 01, 2022 @07:36PM (#30171926)

I have no problem with concurrent writes and wal mode on a recent build of SQLite. Perhaps you are standing up a new sqlite connection per operation or otherwise sharing between separate processes?

With exclusive access using a single connection in WAL mode, I can insert tens of thousands of rows per second on flash storage no problem.

Something that is not widely announced is the fact that SQLite serializes all writes by default, so you should share the same connection instance as often as possible.

[ Parent | on HN ]

Re:Embeddable, replicated and fault tolerant SQL engine base... by ConnorLeet on Tuesday February 01, 2022 @07:11PM (#30171708)

SQLite does have a branch that solves the concurrent write issue - not sure if this dqlite uses it https://sqlite.org/src/doc/754ad35c/README-server-edition.ht...

[ Parent | on HN ]