As mentioned in other comments, a proper defense against this would be to have a front-line system that anticipates such a situation and actively manages it: puts outstanding requests in a queue (throttling the load) and then errors out on new requests if there's still too many of them coming in.
I would assume that if you left it alone for an hour or so it might eventually unfuck itself, but for production purposes, that counts as dead, especially when you can’t even ssh into it because the memory allocations for your ssh session are also in that gigantic queue for disk bandwidth via swap.
edit I hadn't noticed this at the bottom of the article:
> they could have rewritten their web site code so it didn't send the machines into a many-GB-deep swap fest. They could have done that without getting any hosting people involved. They didn't
1. A throttling scheme of some sort
2. Ignore the possibility, and just crash disastrously if/when it occurs
Is that right? If so, surely that's what we should be talking about here, no? (I see ninkendo mentioned it in their comment too.)
Am I right in thinking the whole problem that the blog-post discusses - avoiding a small pool of servers getting overloaded and blowing up - follows from poorly engineered servers that go with Option 2?
Edit This is what I get for reading the article too quickly. She addresses this quite explicitly:
> they could have rewritten their web site code so it didn't send the machines into a many-GB-deep swap fest. They could have done that without getting any hosting people involved. They didn't
Don't assume it's an easy thing to avoid.
I'm really not sold on this attitude. Ordinary exception handling may be a lot of effort, but if you don't do it, that means your work is sloppy. Same here.
More generally, a program shouldn't ever outright explode, regardless of input. That's not quite the same thing as exit-with-error, but I suppose the distinction is ultimately fuzzy.
> The JVM for example can only kill its process once its memory usage exceeds `Xmx`, but it can't prevent that from happening in the first place
I don't see that the particulars of JVM memory-management have any bearing here. A web server should be capable of detecting when it has reached capacity, and to reject additional requests as appropriate. There's no reason this couldn't be done in Java. From a quick search, this is indeed how modern Java servers behave.
If there are many communicating processes, I can imagine that could complicate things greatly. That's a downside of using many communicating processes.
Why shouldn't the system be able to just drop incoming requests until it's able to handle them? That's what routers do.
Perhaps I'm missing something terribly obvious here, but why would that happen?
I can understand requests being dropped and processing-times worsening, but a full system-wide crash?
edit My bad, I'd missed this in the article:
> they could have rewritten their web site code so it didn't send the machines into a many-GB-deep swap fest. They could have done that without getting any hosting people involved. They didn't, and so now I have a story