I’m not sure there is a one-size-fits-all right solution here - but having a system where decryption keys should not be extractable by software alone (e.g. an HSM, a service such as AWS KMS or the equivalents in other clouds) sounds like it would have been an improvement in this case (though with no post-mortem it’s hard to know exactly what went wrong).

Assuming a layered system, something like Vault transit encryption (perhaps with the master key in an HSM) should keep decryption keys away from front-end machines, though as ever, once you have root and can access the memory of a machine where the encryption keys are stored, most bets are off.