News for hackers. Stuff that matters.
Re:Garage – An S3 object store so reliable you can run it ou... by lxpz on Friday December 19, 2025 @02:42PM (#46329988)
If you know of an embedded key-value store that supports transactions, is fast, has good Rust bindings, and does checksumming/integrity verification by default such that it almost never corrupts upon power loss (or at least, is always able to recover to a valid state), please tell me, and we will integrate it into Garage immediately.
Re:Garage – An S3 object store so reliable you can run it ou... by fabian2k on Friday December 19, 2025 @03:50PM (#46330734)
I don't really know enough about the specifics here. But my main points isn't about checksums, but more something like WAL in Postgres. For an embedded KV store this is probably not the solution, but my understanding is that there are data structures like LSM that would result in similar robustness. But I don't actually understand this topic well enough.

Checksumming detects corruption after it happened. A database like Postgres will simply notice it was not cleanly shut down and put the DB into a consistent state by replaying the write ahead log on startup. So that is kind of my default expectation for any DB that handles data that isn't ephemeral or easily regenerated.

But I also likely have the wrong mental model of what Garage does with the metadata, as I wouldn't have expected that to be ever limited by Sqlite.

Re:Garage – An S3 object store so reliable you can run it ou... by lxpz on Friday December 19, 2025 @04:04PM (#46330914)
So the thing is, different KV stores have different trade-offs, and for now we haven't yet found one that has the best of all worlds.

We do recommend SQLite in our quick-start guide to setup a single-node deployment for small/moderate workloads, and it works fine. The "real world deployment" guide recommends LMDB because it gives much better performance (with the current status of Garage, not to say that this couldn't be improved), and the risk of critical data loss is mitigated by the fact that such a deployment would use multi-node replication, meaning that the data can always be recovered from another replica if one node is corrupted and no snapshot is available. Maybe this should be worded better, I can see that the alarmist wording of the deployment guide is creating quite a debate so we probably need to make these facts clearer.

We are also experimenting Fjall as an alternate KV engine based on LSM, as it theoretically has good speed and crash resilience, which would make it the best option. We are just not recommending it by default yet, as we don't have much data to confirm that it works up to these expectations.

Re:Garage – An S3 object store so reliable you can run it ou... by agavra on Friday December 19, 2025 @03:13PM (#46330338)
Sounds like a perfect fit for https://slatedb.io/ -- it's just that (an embedded, rust, KV store that supports transactions).

It's built specifically to run on object storage, currently relies on the `object_store` crate but we're consdering OpenDAL instead so if Garage works with those crates (I assume it does if its S3 compatible) it should just work OOTB.

Re:Garage – An S3 object store so reliable you can run it ou... by evil-olive on Saturday December 20, 2025 @02:42PM (#46338914)
for Garage's particular use case I think SlateDB's "backed by object storage" would be an anti-feature. their usage of LMDB/SQLite is for the metadata of the object store itself - trying to host that metadata within the object store runs into a circular dependency problem.
Re:Garage – An S3 object store so reliable you can run it ou... by patmorgan23 on Friday December 19, 2025 @04:11PM (#46330980)
Valkey?
Re:Garage – An S3 object store so reliable you can run it ou... by BeefySwain on Friday December 19, 2025 @02:47PM (#46330055)
(genuinely asking) why not SQLite by default?
Re:Garage – An S3 object store so reliable you can run it ou... by lxpz on Friday December 19, 2025 @02:54PM (#46330120)
We were not able to get good enough performance compared to LMDB. We will work on this more though, there are probably many ways performance can be increased by reducing load on the KV store.
Re:Garage – An S3 object store so reliable you can run it ou... by tensor on Friday December 19, 2025 @03:46PM (#46330707)
Keep in mind that write safety comes with performance penalties. You can turn off write protections and many databases will be super fast, but easily corrupt.
Re:Garage – An S3 object store so reliable you can run it ou... by skrtskrt on Friday December 19, 2025 @03:22PM (#46330439)
Could you use something like Fly's Corrosion to shard and distribute the SQLite data? It uses a CRDT reconciliation, which is familiar for Garage.
Re:Garage – An S3 object store so reliable you can run it ou... by lxpz on Friday December 19, 2025 @03:44PM (#46330681)
Garage already shards data by itself if you add more nodes, and it is indeed a viable path to increasing throughput.
Re:Garage – An S3 object store so reliable you can run it ou... by srcreigh on Friday December 19, 2025 @04:49PM (#46331352)
Did you try WITHOUT ROWID? Your sqlite implementation[1] uses a BLOB primary key. In SQLite, this means each operation requires 2 b-tree traversals: The BLOB->rowid tree and the rowid->data tree.

If you use WITHOUT ROWID, you traverse only the BLOB->data tree.

Looking up lexicographically similar keys gets a huge performance boost since sqlite can scan a B-Tree node and the data is contiguous. Your current implementation is chasing pointers to random locations in a different b-tree.

I'm not sure exactly whether on disk size would get smaller or larger. It probably depends on the key size and value size compared to the 64 bit rowids. This is probably a well studied question you could find the answer to.

[1]: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/4efc8...

Re:Garage – An S3 object store so reliable you can run it ou... by lxpz on Friday December 19, 2025 @05:09PM (#46331565)
Very interesting, thank you. It would probably make sense for most tables but not all of them because some are holding large CRDT values.
Re:Garage – An S3 object store so reliable you can run it ou... by asa400 on Saturday December 20, 2025 @05:19PM (#46340175)
Other than knowing this about SQLite beforehand, is there any way one could discover that this is happening through tracing?
Re:Garage – An S3 object store so reliable you can run it ou... by rapnie on Friday December 19, 2025 @07:10PM (#46332483)
I learned that Turso apparently have plans for a rewrite of libsql [0] in Rust, and create a more 'hackable' SQLite alternative altogether. It was apparently discussed in this Developer Voices [1] video, which I haven't yet watched.

[0] https://github.com/tursodatabase/libsql

[1] https://www.youtube.com/watch?v=1JHOY0zqNBY

Re:Garage – An S3 object store so reliable you can run it ou... by __turbobrew__ on Friday December 19, 2025 @04:38PM (#46331259)
RocksDB possibly. Used in high throughput systems like Ceph OSDs.
Re:Garage – An S3 object store so reliable you can run it ou... by johncolanduoni on Saturday December 20, 2025 @12:43AM (#46333900)
I’ve used RocksDB for this kind of thing in the past with good results. It’s very thorough from a data corruption detection/rollback perspective (this is naturally much easier to get right with LSMs than B+ trees). The Rust bindings are fine.

It’s worth noting too that B+ tree databases are not a fantastic match for ZFS - they usually require extra tuning (block sizes, other stuff like how WAL commits work) to get performance comparable to XFS/ext4. LSMs on the other hand naturally fit ZFS’s CoW internals like a glove.

Re:Garage – An S3 object store so reliable you can run it ou... by VerifiedReports on Saturday December 20, 2025 @01:06AM (#46333975)
It's "key/value store", FYI
Re:Garage – An S3 object store so reliable you can run it ou... by abustamam on Saturday December 20, 2025 @01:16AM (#46334008)
Wikipedia seems to find "key-value store" an appropriate term.

https://en.wikipedia.org/wiki/Key%E2%80%93value_database

Re:Garage – An S3 object store so reliable you can run it ou... by VerifiedReports on Saturday December 20, 2025 @12:59PM (#46338087)
See above.
Re:Garage – An S3 object store so reliable you can run it ou... by abustamam on Monday December 22, 2025 @11:35AM (#46355601)
Still not sure what point you're trying to make. You attempted to correct GP's usage of "key-value store" and I merely pointed out that it is the widely accepted term for what is being discussed.

Whether or not it's semantically "correct" because of usage of hyphen vs slash is irrelevant to that point.

Re:Garage – An S3 object store so reliable you can run it ou... by kqr on Saturday December 20, 2025 @01:26AM (#46334041)
It's not a store of "keys or values", no. It's a store of key-value pairs.
Re:Garage – An S3 object store so reliable you can run it ou... by VerifiedReports on Saturday December 20, 2025 @12:59PM (#46338085)
A key-value store would be a store of one thing: key values. A hyphen combines two words to make an adjective, which describes the word that follows:

  A used-car lot

  A value-added tax

  A key-based access system
When you have two exclusive options, two sides to a situation, or separate things; you separate them with a slash:

  An on/off switch

  A win/win situation

  A master/slave arrangement
Therefore a key-value store and a key/value store are quite different.
Re:Garage – An S3 object store so reliable you can run it ou... by kqr on Saturday December 20, 2025 @02:02PM (#46338595)
All of your slash examples represent either–or situations. A swich turns it on or off, the situation is a win in the first outcome or a win in the second outcome, etc.

It's true that key–value store shouldn't be written with a hyphen. It should be written with an en dash, which is used "to contrast values or illustrate a relationship between two things [... e.g.] Mother–daughter relationship"

https://en.wikipedia.org/wiki/Dash#En_dash

I just didn't want to bother with typography at that level of pedanticism.

Re:Garage – An S3 object store so reliable you can run it ou... by VerifiedReports on Saturday December 20, 2025 @03:01PM (#46339073)
No, they don't. A master/slave configuration (of hard drives, for example) involves two things. I specifically included it to head off the exact objection you're raising.

"...the slash is now used to represent division and fractions, as a date separator, in between multiple alternative or related terms"

-Wikipedia

And what is a key/value store? A store of related terms.

And if you had a system that only allowed a finite collection of key values, where might you put them? A key-value store.

Re:Garage – An S3 object store so reliable you can run it ou... by kqr on Saturday December 20, 2025 @03:03PM (#46339094)
The hard drives are either master or slave. A hard drive is not a master-and-slave.
Re:Garage – An S3 object store so reliable you can run it ou... by VerifiedReports on Sunday December 21, 2025 @12:43AM (#46342533)
Exactly. And an entry in a key/value store is either a key or a value. Not both.
Re:Garage – An S3 object store so reliable you can run it ou... by kqr on Sunday December 21, 2025 @01:38AM (#46342771)
No, an entry is a key-and-value pair. Are you deriously suggesting it is possible to add only keys without corresponding values, or vice versa?
Re:Garage – An S3 object store so reliable you can run it ou... by DonHopkins on Saturday December 20, 2025 @10:48AM (#46336980)
Which is infinite of value is zero.
Re:Garage – An S3 object store so reliable you can run it ou... by __padding on Saturday December 20, 2025 @07:29PM (#46341099)
I’ve not looked at it in a while but sled/rio were interesting up and coming options https://github.com/spacejam/sled
Re:Garage – An S3 object store so reliable you can run it ou... by ndyg on Sunday December 21, 2025 @12:20AM (#46342443)