Re:Hctree is an experimental high-concurrency database back ...
by tptacek on Wednesday January 18, 2023 @09:43PM
(#34436120)
I spent a bunch of time wrestling with this last year (and into this one), and the general solution I settled on was to serialize writes through a single writer; multiple reader threads and multiple writer threads (or processes) is not a fun configuration to put SQLite in.
There were 500MB-40G databases, probably ~1.5 updates per second at peak load?
Re:Hctree is an experimental high-concurrency database back ...
by sqlcommando on Thursday January 19, 2023 @12:19AM
(#34436977)
I think "web app" is too broad a category to have this kind of discussion over.. for a lot of web apps 1.5 updates per second could be considered a reasonable limit. Many people work on web apps that do 100x that and don't consider themselves to be working at any particular level of scale. 1.5K updates/sec is not uncommon either, in different settings -- high-traffic internal web apps can hit that in a single company, and the people working in those contexts don't consider that too special either, not to mention public facing websites like GH where many people start their careers without having seen anything different.
Re:Hctree is an experimental high-concurrency database back ...
by tptacek on Thursday January 19, 2023 @12:44AM
(#34437113)
I'm not benchmarking SQLite, I'm relating the environment in which I was tuning SQLite concurrency, because I feel like it doesn't mean much to say "here's what worked for me" without at least some basic facts about what I was working on. SQLite itself can do tens of thousands of writes per second on commodity hardware without breaking a sweat.
Having one writer and multiple readers was fine, but even having a dozen writers who update something a few times per second quickly led to the DB becoming the bottleneck. I tried different modes, but saw no difference so I just gave up on using it concurrently.