News for hackers. Stuff that matters.
Re:Distributed, high-performance blob storage on SQLite by anacrolix on Thursday November 19, 2026 @07:05AM (#50559043)
I spent years trying to squeeze this use case out of sqlite. All roads lead to tears, it is not made for this use case.

I wrote https://github.com/anacrolix/possum for this exact use case. It has a few rough edges but it absolutely hauls ass for caching on disk. Incidentally it uses sqlite for transaction/TTL metadata which is more suited to sqlite but it's still the bottleneck.

sqlite transaction commit overhead is too high and uncompetitive without batching but it's a design flaw specific to being multiprocess.

Re:Distributed, high-performance blob storage on SQLite by mightyham on Thursday November 19, 2026 @07:54AM (#50559490)
At the end of the day durable transaction commit in any OLTP database will always be slow because fsync is an expensive operation (~1000 IOPS on a decent SSD), so as you mentioned batching is the only way to make significant increases to throughput. That said, based on assorted benchmarks I've seen online, sqlite doesn't even get close to max fsync IOPS because it has a ton of overhead. Other embedded systems like rocksdb or lmdb are going to give better performance.
Re:Distributed, high-performance blob storage on SQLite by luke5441 on Thursday November 19, 2026 @09:13AM (#50560379)
Or you can do like GP with that possum and not fsync at all. It even sets synchronous=OFF, which will corrupt SQLite databases. Then it is fast because it only writes to memory. YOLO!
Re:Distributed, high-performance blob storage on SQLite by stingraycharles on Thursday November 19, 2026 @09:25AM (#50560537)
The usual answer here is a WAL + batching multiple transactions.
Re:Distributed, high-performance blob storage on SQLite by adev_ on Thursday November 19, 2026 @10:05AM (#50561069)
> Incidentally it uses sqlite for transaction/TTL metadata which is more suited to sqlite but it's still the bottleneck.

Why sqlite instead of RocksDB ?

What you need at the end of the day for a blob storage is a transactional key value store.

Which RocksDB will give you with much better throughput (and latency) and solves your transaction problem.

Re:Distributed, high-performance blob storage on SQLite by mbreese on Thursday November 19, 2026 @11:48AM (#50562553)
Yeah, I'm not sure how well this will work. I've always dealt with data blobs with data on the filesystem (or s3), metadata in the database. Anything else starts to get a bit too much like a filesystem in a database, which has lead many smart people to tears.

Or is there a better way?

Re:Distributed, high-performance blob storage on SQLite by UltraSane on Thursday November 19, 2026 @12:22PM (#50563015)
Metadata in a DB or index of some kind and the actual data in a key-val object store is the most versatile configuration I have used. This lets you use as many different metadata indexes as you need.
Re:Distributed, high-performance blob storage on SQLite by senderista on Thursday November 19, 2026 @12:36PM (#50563226)
> the actual data in a key-val object store

So, on a filesystem?

Re:Distributed, high-performance blob storage on SQLite by UltraSane on Thursday November 19, 2026 @01:16PM (#50563806)
You can use one if you don't need the scale of a true object storage system.