Re:Building a highly-available web service without a database
by sahil-kang on Saturday August 10, 2024 @07:24PM
(#41212962)
fwiw, I normally think of LMDB when the idea of using the fs instead of a db crosses my mind
Re:Building a highly-available web service without a database
by djhn on Sunday August 11, 2024 @01:55AM
(#41214245)
Why not sqlite? put the json in a single column, maybe copy some parts of it or metadata to another two or three. Should be faster than the filesystem for reading multiple rows.
Re:Building a highly-available web service without a database
by lpapez on Sunday August 11, 2024 @03:38AM
(#41214569)
Why not start with SQLite?
There are benchmarks out there proving that for some use cases (i.e. many small updates) where using SQLite is faster than using the filesystem. [1]
So not only do you get all of the benefits of a relational database, and literally centuries of engineering hours and bugfixes invested into SQLite, you might also get better performance (which is why I presume you even considered rolling your own).
I did this sort of thing recently. I felt bad doing it, I still objectively hate it, because I do know enough to know that basically I'm re-implementing what years of hardworking O/S developers have done, piecemeal. But at least I'm going in with my eyes open which feels better.
The only real mitigating factor I have is that the application is largely 'never-read' and then when reading is done, it's sequential batches. Which is not normally something databases optimise for and works okay for file-storage.
(If someone does know a lightweight database architecture that performs like this, let me know).