Re:The GitHub Availability Report
by dx034 on Thursday July 16, 2020 @03:29AM
(#23856687)
What confuses me is that CPU starvation caused crashes. Sure, too few CPU ressources should cause a slow down but the database process itself shouldn't crash as a result. I haven't dealt with MySQL that much but at least for Postgres, I've never seen 100% load to cause a node to fail.
[ Parent
| on HN ]
I could see a buffer for requests filling up too much so that it causes a no memory crash.
But even that seems preventable.
OOM issues often don't _really_ cause a crash unless you are running with real memory only. Usually long before you are out of total memory you are using so much virtual memory that the kernel is thrashing IO to try keep up with all the page faults. At this point the machine hasn't crashed, but from the outside it might as well have because it can't respond to anything new, even diagnostic intervention, in a timely fashion.
CPU overloading, without memory starvation, can have a similar effect, though that doesn't tend to fall off the same efficiency wall as when IO is the issue (unless something has literally kicked off a fork bomb) because of the orders of magnitude differences in latency and work throughput.
A swap livelock would be worse than a crash, you really don't want a DB backend to start swapping in this kind of application.
No, but from a diagnostics PoV it can be important to know the difference between failure modes that, unless you are physically near the device so can see drive lights flickering (or hear heads moving if using traditional drives), could otherwise look identical.