Re:Distributed, high-performance blob storage on SQLite
by mightyham on Thursday November 19, 2026 @07:54AM
(#50559490)
At the end of the day durable transaction commit in any OLTP database will always be slow because fsync is an expensive operation (~1000 IOPS on a decent SSD), so as you mentioned batching is the only way to make significant increases to throughput. That said, based on assorted benchmarks I've seen online, sqlite doesn't even get close to max fsync IOPS because it has a ton of overhead. Other embedded systems like rocksdb or lmdb are going to give better performance.
Re:Distributed, high-performance blob storage on SQLite
by luke5441 on Thursday November 19, 2026 @09:13AM
(#50560379)
Or you can do like GP with that possum and not fsync at all. It even sets synchronous=OFF, which will corrupt SQLite databases.
Then it is fast because it only writes to memory. YOLO!
Re:Distributed, high-performance blob storage on SQLite
by stingraycharles on Thursday November 19, 2026 @09:25AM
(#50560537)
The usual answer here is a WAL + batching multiple transactions.
Re:Distributed, high-performance blob storage on SQLite
by adev_ on Thursday November 19, 2026 @10:05AM
(#50561069)
> Incidentally it uses sqlite for transaction/TTL metadata which is more suited to sqlite but it's still the bottleneck.
Why sqlite instead of RocksDB ?
What you need at the end of the day for a blob storage is a transactional key value store.
Which RocksDB will give you with much better throughput (and latency) and solves your transaction problem.
Re:Distributed, high-performance blob storage on SQLite
by mbreese on Thursday November 19, 2026 @11:48AM
(#50562553)
Yeah, I'm not sure how well this will work. I've always dealt with data blobs with data on the filesystem (or s3), metadata in the database. Anything else starts to get a bit too much like a filesystem in a database, which has lead many smart people to tears.
Or is there a better way?
Re:Distributed, high-performance blob storage on SQLite
by UltraSane on Thursday November 19, 2026 @12:22PM
(#50563015)
Metadata in a DB or index of some kind and the actual data in a key-val object store is the most versatile configuration I have used. This lets you use as many different metadata indexes as you need.
Re:Distributed, high-performance blob storage on SQLite
by senderista on Thursday November 19, 2026 @12:36PM
(#50563226)
> the actual data in a key-val object store
So, on a filesystem?
I wrote https://github.com/anacrolix/possum for this exact use case. It has a few rough edges but it absolutely hauls ass for caching on disk. Incidentally it uses sqlite for transaction/TTL metadata which is more suited to sqlite but it's still the bottleneck.
sqlite transaction commit overhead is too high and uncompetitive without batching but it's a design flaw specific to being multiprocess.