News for hackers. Stuff that matters.
Re:Hctree is an experimental high-concurrency database back ... by simonw on Wednesday January 18, 2023 @08:50PM (#34435761)
Most SQLite writes can be measured in microseconds. If your app is doing less than 1,000 writes a second you probably won't even notice that SQLite is serializing your writes rather than applying them concurrently.
Re:Hctree is an experimental high-concurrency database back ... by luckylion on Wednesday January 18, 2023 @09:41PM (#34436115)
I must've been doing something wrong then, I've regularly hit longer locks as soon as the DB grew, to the point that it was not usable for me with multiple processes writing in the same DB because it was constantly locked by one.

Having one writer and multiple readers was fine, but even having a dozen writers who update something a few times per second quickly led to the DB becoming the bottleneck. I tried different modes, but saw no difference so I just gave up on using it concurrently.

Re:Hctree is an experimental high-concurrency database back ... by tptacek on Wednesday January 18, 2023 @09:43PM (#34436120)
I spent a bunch of time wrestling with this last year (and into this one), and the general solution I settled on was to serialize writes through a single writer; multiple reader threads and multiple writer threads (or processes) is not a fun configuration to put SQLite in.

There were 500MB-40G databases, probably ~1.5 updates per second at peak load?

Re:Hctree is an experimental high-concurrency database back ... by sqlcommando on Thursday January 19, 2023 @12:19AM (#34436977)
I think "web app" is too broad a category to have this kind of discussion over.. for a lot of web apps 1.5 updates per second could be considered a reasonable limit. Many people work on web apps that do 100x that and don't consider themselves to be working at any particular level of scale. 1.5K updates/sec is not uncommon either, in different settings -- high-traffic internal web apps can hit that in a single company, and the people working in those contexts don't consider that too special either, not to mention public facing websites like GH where many people start their careers without having seen anything different.
Re:Hctree is an experimental high-concurrency database back ... by tptacek on Thursday January 19, 2023 @12:44AM (#34437113)
I'm not benchmarking SQLite, I'm relating the environment in which I was tuning SQLite concurrency, because I feel like it doesn't mean much to say "here's what worked for me" without at least some basic facts about what I was working on. SQLite itself can do tens of thousands of writes per second on commodity hardware without breaking a sweat.
Re:Hctree is an experimental high-concurrency database back ... by ec109685 on Thursday January 19, 2023 @03:14AM (#34437919)
There's more to most applications than interactions with the database. Once your web app exceeds the capacity of a single machine, is SQLite a good choice for state?