Periodically we would run a full sync to replace the database. Between the periodic full syncs, we had a background process keep changes applied on a rolling basis.
All-in-all, it worked pretty well at the time! The full database file sync ensured a bad database was timeboxed and we got a bootstrapping mechanism for free.
I'm not knowledgeable on this, but my understanding was a b-tree is a way of sorting values that could be ordered in a certain way. Like this would be a b-tree of IDs
```
[8]
/ \
[3, 5] [10, 12]
/ | \ / | \
[1] [4] [6,7] [9] [11, 13]
```You traverse by comparing your needle to the root node and going left or right depending on the results.
How is that done with configuration options? That seems like it would just be a regular hashmap which is already efficient to read. What would a b-tree of key/values even look like that wouldn't be less efficient than a hashmap?
Databases including SQLite usually use b+tree for tables (a variant where only the leaves have data, the interior nodes only have keys) and regular btrees for indexes.
So it might seem that simply enumerating the data (sorted or not) would be a better option for a file. (After all, the receiver will read everything anyway.) I guess frequent updates make this inefficient, so a tree helps.
Now, that's a name I've not heard in a long time.
Are people still using Kyoto Cabinet in new projects? Are people still using DBM-style storage generally? I thought that whole branch of the evolutionary tree had sort of died out.
Cloudflare used to use Kyoto Cabinet[1] and moved to LMDB[1] in 2020; other implementations that I'm familiar with (but don't have a link to share) also use LMDB.
> Are people still using DBM-style storage generally?
It's fairly common in these scenarios, as well as the underlying key-value store for popular software like Consul[3].
[1] https://blog.cloudflare.com/kyoto-tycoon-secure-replication/
[2] https://blog.cloudflare.com/introducing-quicksilver-configur...
https://blog.cloudflare.com/moving-quicksilver-into-producti...
In any case, SQLite would serve this solution, but almost certainly with a measurable level of inefficiency built in.
Edit: @akira2501: SQLite comes with different tradeoffs, sometimes superior and other times less so, depending on the use case. Blanket statements without evidence are unproductive in progressing the conversation..
https://www.sqlite.org/optoverview.html
Chapter 2. Chapter 7. Chapter 8.
It should be _zero_ surprise to you that SQLite is not as powerful as other full database query engines.
But then it was quite fast.
But as with all things SQLite, the answer is generally "it's easy to predict so just improve your query" and/or "there's a plugin for that" so it tends to win in the end with a bit of care.
SQLite's planner is anything but simple. It has a beam search over different plans, basically, it approximates full NP-hard solution search process using bounded space and time.
BTW, this is also the model used by all CDNs, where the global configuration file containing the certificates, HTTP routing rules etc. for all customers will be updated into into a single-file b-tree structure*, and that "bundle" is distributed among all edge locations frequently.
* I'm yet to see someone use sqlite for this purpose, it's usually DBM style databases like LMDB or Kyoto Cabinet.