Segmentation can help. For example, you can use envelope encryption[1] that keeps your keys on an isolated, dedicated key management server that prevents even the admin from exporting the master keys. Therefore, decryption of the data keys must be performed on the key management server.

It’s still not perfect because an attacker can potentially send requests to the key management server, but they shouldn’t be able to walk away with they keys to perform decryption outside of the system.

You then set up monitoring to watch how many decryption operations are being performed per minute or hour and alert admins when it steps outside of normal useage patterns.

It’s not perfect but can help you to 1) catch an attacker early and 2) have some type of estimate of the size of the breach when discovered. A very patient attacker may not trigger an alarm of decryption operations but in that case he’s got to work much slower, which limits the scope of the attack while they hopefully trigger something else that exposes them.

1: https://devender.me/2016/07/13/envelope-encryption/

Have a server responsible solely for decryption and audit access to it. The server doesn’t issue keys, it literally does the decryption. AWS KMS can do this for you.

The gist is you create an encryption key for your row, encrypt it using your encryption service, and store it next to your actual payload. To decrypt, ask the service to decrypt the key which you use to decrypt the payload. If your database gets popped, your decryption server hopefully didnt because you hardened it specifically.

Well, it's still tough. The question is always "gained root to what", cause at some point somebody needs a key.

Envelope encryption is a good scheme. Under envelope encryption you have a master key and per-row or per-unit keys. You store the per-unit keys encrypted with the master key, and the master key is stored offsite.

For example on an AWS environment, you would use AWS KMS with IAM authentication to handle the master key crypt for the individual keys, and the encrypted versions of the individual keys are stored in a database that you own.

You encrypt your data with the individual keys (eg, one key per user, or one key per DB row, or one key per namespace, whatever), but the individual keys are decryptable only by KMS.

Under this scheme an attacker must both be able to get ahold of the encrypted individual keys, and be able to decrypt them with the master key.

Of course this leaves you very vulnerable if the master key is acquired, but KMS does not allow you to read the master key at all, you have to use the AWS API to make encryption/decryption requests. So the vulnerability is less about how secure the key is and more about how secure your IAM setup and instances are.

The application necessarily has access to the credentials it needs to do all of this. You're only eliminating the subset of attackers who can get in, but not figure out how to imitate your use of cryptography. Is the cryptography part even relevant? Since the attacker ultimately does have use of the key material, there's no information-theoretic security provided by the encryption, only some extra work to do. Wouldn't any other indirection/obfuscation accomplish the same goal?

Or are we assuming that the decryption service enforces some policy about what and how much to decrypt?

Well, yes, if you're using AWS KMS you are able to set up policies, revoke access through IAM, use alerts and monitoring, and so on.

But you're also forgetting that most data breaches in real life involve database backups, staging servers, development environments, and people with access. So envelope encryption really helps prevent those. Even if someone gets a hand on your DB dump they're unable to use it without authorization to the KMS, which they won't have unless they can also get into your application server.

I guess I’m missing how credentials with access to KMS are any harder to come by than credentials with access to RDS, and why it’s easier to monitor KMS queries than RDS queries. Are the latter privileges more widely distributed, maybe? Where is it useful to have access to a database you can’t actually read? Do your dev/staging environments just no-op the decryption and put ciphertext strings everywhere prod shows plaintext?

Maybe your DB creds are in a secure configuration system and maybe the KMS credentials are granted by an IAM instance role. If only certain users are able to get access to config (ie RDS), and only certain machines are able to decrypt data, you now have a multilayered security approach where an attacker needs to breach both a user and a piece of hardware. If you breach a user and get your hands on a DB leak, you can't use it because you don't have access to the hardware. If you breach the instance role then you can make decrypt requests, but you would have also needed to have breached the data separately and independently.

Don't forget that data is most often leaked by people. DB backups, dev environments, inappropriate transfers. We hear about the big targeted attacks, but probably 90% of actual breaches are due to human misuse.

Depends on the use case :)

If you don't need to decrypt the data on the same system, you could always use asymmetric encrpytion, so encrypt with the "public" key and then keep the private key elsewhere.

If you require to be able to decrypt on the system in question then you need a key somewhere, so secure storage of the key is very important.

One option, often deployed in banks, is to store the key in a hardware security module (HSM) and then ensure that it's not available outside of that.

There's a load of tradeoffs to using HSMs but they can be useful where symmetric encrpytion is needed.

This was solve by banks a long time ago with hardware security modules (https://en.m.wikipedia.org/wiki/Hardware_security_module). HSMs make it impossible to extract encryption keys from the device and turn a digital security problem into a physical security problem which banks knew how to manage. Are your encryption keys secure? Yes they are right there.

Amazon and Azure offer these in the cloud.

The application has whatever credentials it needs to make decryption requests to the HSM. Wouldn't an attacker who owns it simply find them?

Typically the HSM will be part of a tokenization service that is located in a separate security zone also called a data bunker or data vault. The customer facing application will pass in the sensitive data and in its place a token will be returned. When the calling application needs the data it will pass in the token and the data is returned.

An attacker that compromised the customer facing application only has access to tokens and would be forced to access sensitive data via the data bunker. That application is heavily monitored and would hopefully be quickly noticed.

The HSM adds an additional layer of protection against loss of sensitive data via backup tapes or some insider attacks.

It's been several years since I dealt with PCI, but the app I was on the team for handled $1B+ annually. Here's just a few points that I remember.

Physically separate the key storage from the data storage. Different servers, different user auth mechanism. Make them compromise both systems.

Unique keys per client.

Use key-encrypting keys, periodically replace them.

PCI environment must be totally self contained and not mixed with standard working environment. If an employee is phished, their credentials for email/etc should not gain access to the env.

Don't allow developers to deploy or even read from prod db. Have a separate deployment team, all access to prod database audited via production tickets.

I’m not sure there is a one-size-fits-all right solution here - but having a system where decryption keys should not be extractable by software alone (e.g. an HSM, a service such as AWS KMS or the equivalents in other clouds) sounds like it would have been an improvement in this case (though with no post-mortem it’s hard to know exactly what went wrong).

Assuming a layered system, something like Vault transit encryption (perhaps with the master key in an HSM) should keep decryption keys away from front-end machines, though as ever, once you have root and can access the memory of a machine where the encryption keys are stored, most bets are off.

Something not widely used are canary tokens: https://blog.thinkst.com/p/canarytokensorg-quick-free-detect...

The general idea is to insert some kind of bogus information into a system, with no links to other systems, and then trigger a notification if someone accesses it.

One tool is to only get access to this data from the Web FE via an API. This API is then monitored for spikes in usage, so if a given FE is suddenly requesting a lot more than normal it gets flagged. It doesn't stop targeted attacks, but does show the big ones - another reason why security needs to be multi layered.

Keep in mind they've been compromised for 4 years. For that time duration it's not just about protecting data at rest, it's also about protecting data as it is processed inside your systems.

>hackers when they have essentially gained root?

You are already fucked at this point. I would focus my attention on prevent hackers from 1) gaining any unauthorized access. 2) from doing any sort of privilege escalation from a restricted account/service.

Maybe keep the keys only in memory in each application? Pull them from some very well monitored service once at startup? Anyone who gained access to the server would have to check the program's memory to get them.