I work for another shop that scolds me for even suggesting using something else than k8s. How can a persistent file system possibly scale? Lol. Ironically they're grossing -$3 million/year.
Sqlite is a disk based database...
The main problem I was uneasy about was, I would simply use the equivalent of rsync -Pa /ramdisk /disk in a loop, which seemed to ensure safety in the event of a poweroff. It never broke, but I wouldn’t say it was a good idea, necessarily.
Sure speeds up most things though. The benefits are pretty crazy for I/O bound programs.
That's probably a bad idea. rsync isn't going to copy over writes in the same order they were written by the DB. Most DBs assume that when there's some crash that at least the disk would have been in a logically consistent state with respect to the order of certain writes separated by e.g. fsync and barriers.
Randomly inject bit flips into a target process memory space, so you can see/test how your software handles it (if at all).
The IBM 53-bit quantum compute result from a ton of hard drives + compute power was also intriguing (https://www.ibm.com/blogs/research/2019/10/on-quantum-suprem...).
128PiB of disk space needed for a 54-qubit computation on their system (hard drives). Fun to see where the supercomputers are still needed, but smaller qubits can be emulated on normal computers. 40 qubit calculations probably can be emulated on a 16TB team of hard drives in RAID0.
Ninja edit: or a UPS and a shutdown procedure triggered by it that writes the database to disk. You can write 100GB in a couple minutes over a 10Gbps network. Faster to a local SSD.
There are just certain classes of data (e.g. any kind of business transaction) where you can’t afford this type of best effort durability.
This is another in the "Now that we have computers with gazillions of bits of RAM and many cores and gazillions of cycles, what conventional wisdom is no longer wise?" series :-).
I suggested to a client that they use Sqlite3 for their company database rather than a disk based database. And they looked at me shocked, as if I had spoken heresy. I pointed out how many records they could store on a 64GB machine and that is a small data center class machine and it sort of boggled their mind. Add read-only copies, shared from a NAS device gave them perfect scalability in terms of queries per second (their sales folks had their own database that they were updating). It was fun to watch their expression go from horror to wonder to excitement when they finally implemented it.