After swapping out the SD card with a new one, it appears to be fixed. This was hard to diagnose as there was nothing in system logs / error messages about what might have went wrong.
- don't use swap
- make sure you have noatime on all your filesystems
- identify other folders with frequently changing data (/tmp, $HOME/tmp, etc.) and also make sure you keep them in memory
- you can use NFS for many of these things if that is possible
Why is this not the default?
So many people struggle with the same stuff over and over again, yet, there's no mechanism in Linux distros to package default settings for a use case.
What they ought to do is put an easy way to set that in `raspi-config`
Typically things are geared for desktop users who have not much idea what they are doing and everybody else is expected to understand and configure to their needs.
Nowadays distributions started preparing separate profiles (workstation, desktop, embedded, etc.) that save a little bit of work.
noatime is a default for most distributions but a person that does install on Raspberry Pi might be configuring filesystem manually and not be aware to put it there.
Swap is very tempting to have on the Pi but is going to kill the card in a hurry.
Therefore I wouldn’t recommend the rpi anymore.
Ideally the next Pi gets some interface that allows more integrated storage, like eMMC or NVMe, but even with USB the speed and durability justifies annoying cabling situations.
Even if it does wear out, storing all the data on NFS (or a usb drive) makes it a lot less worrying.