My ignorance may be showing, how would I copmpletely isolate the private keys from the public-facing service? I suppose using an accelerator card would do it?
You run an "RSA signing server" that accepts connections only from the internal IP of your webserver (or maybe only TLS connections from across the internet but only if the client connecting to it has a client certificate signed by your own self-signed CA, but that depends on topology).
It runs a very simple app that accepts requests to sign something and responds with the <something> signed by your RSA private key. The code for this is very minimal and secure. You teach your web server to use this thing during TLS handshake, to sign the ephemeral key exchange.
If the web server is hacked, the hackers gain the ability to sign things with your private key, but they don't get the private key itself. They need to hack the "RSA signing service" server for that. They can't stockpile signed ephemeral key exchanges. After you detect the hack, snapshot and kill the hacked server, start a new one from clean backup, they lose the ability to impersonate your site.
CloudFlare have a writeup on their service that does this (TLS negotiation by their TLS terminator with only you holding the key): https://blog.cloudflare.com/keyless-ssl-the-nitty-gritty-tec...
BTW, you can store private keys in a HSM on your private key server and that's an additional layer of protection, but nobody uses TLS accelerator cards anymore that I'm aware of. You can simply do the relevant crypto on commodity CPUs and still have plenty of throughput.
1. Store your private keys in a separate process, either on the same machine or a remote, more secure host. Offload private key operations to the private key service. This obviously requires an encrypted connection between your webserver and the private key service, but you get good gains in security from no longer having the private key in your public facing web server's address space.
2. Another technique you can add is to separate the data plane from the control plane on your public facing webserver--that way you have a stripped-down process that is just handling the low level reading and writing of buffers to the wire, and have a fast pipe connecting it to a separate process doing all the HTTP logic. That way, you can lock down the data plane service and make it harder to exploit since it has a much smaller surface area than a full fledged web server.