News for hackers. Stuff that matters.
Re:Garage – An S3 object store so reliable you can run it ou... by lxpz on Friday December 19, 2025 @02:54PM (#46330120)
We were not able to get good enough performance compared to LMDB. We will work on this more though, there are probably many ways performance can be increased by reducing load on the KV store.
Re:Garage – An S3 object store so reliable you can run it ou... by tensor on Friday December 19, 2025 @03:46PM (#46330707)
Keep in mind that write safety comes with performance penalties. You can turn off write protections and many databases will be super fast, but easily corrupt.
Re:Garage – An S3 object store so reliable you can run it ou... by skrtskrt on Friday December 19, 2025 @03:22PM (#46330439)
Could you use something like Fly's Corrosion to shard and distribute the SQLite data? It uses a CRDT reconciliation, which is familiar for Garage.
Re:Garage – An S3 object store so reliable you can run it ou... by lxpz on Friday December 19, 2025 @03:44PM (#46330681)
Garage already shards data by itself if you add more nodes, and it is indeed a viable path to increasing throughput.
Re:Garage – An S3 object store so reliable you can run it ou... by srcreigh on Friday December 19, 2025 @04:49PM (#46331352)
Did you try WITHOUT ROWID? Your sqlite implementation[1] uses a BLOB primary key. In SQLite, this means each operation requires 2 b-tree traversals: The BLOB->rowid tree and the rowid->data tree.

If you use WITHOUT ROWID, you traverse only the BLOB->data tree.

Looking up lexicographically similar keys gets a huge performance boost since sqlite can scan a B-Tree node and the data is contiguous. Your current implementation is chasing pointers to random locations in a different b-tree.

I'm not sure exactly whether on disk size would get smaller or larger. It probably depends on the key size and value size compared to the 64 bit rowids. This is probably a well studied question you could find the answer to.

[1]: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/4efc8...

Re:Garage – An S3 object store so reliable you can run it ou... by lxpz on Friday December 19, 2025 @05:09PM (#46331565)
Very interesting, thank you. It would probably make sense for most tables but not all of them because some are holding large CRDT values.
Re:Garage – An S3 object store so reliable you can run it ou... by asa400 on Saturday December 20, 2025 @05:19PM (#46340175)
Other than knowing this about SQLite beforehand, is there any way one could discover that this is happening through tracing?
Re:Garage – An S3 object store so reliable you can run it ou... by rapnie on Friday December 19, 2025 @07:10PM (#46332483)
I learned that Turso apparently have plans for a rewrite of libsql [0] in Rust, and create a more 'hackable' SQLite alternative altogether. It was apparently discussed in this Developer Voices [1] video, which I haven't yet watched.

[0] https://github.com/tursodatabase/libsql

[1] https://www.youtube.com/watch?v=1JHOY0zqNBY