~"We encrypt sensitive data at rest. And we store decryption keys on our user facing app servers so that apps can decrypt the data. Cool, right?"

Lets be honest, all reversible encryption schemes are some version of this...

I'm not quite sure whether irreversible encryption is a thing in this context (its sometimes used to talk about e.g. password storage, but I think that's a very different use-case), so I'll sally in with an interesting tangent:

https://en.wikipedia.org/wiki/Homomorphic_encryption

   Homomorphic encryption is a form of encryption that allows
   computation on ciphertexts, generating an encrypted result
   which, when decrypted, matches the result of the 
   operations as if they had been performed on the plaintext.

   Homomorphic encryption can be used for secure outsourced
   computation
   ...

That's interesting, but how do you prevent bad guys from "generating an encrypted result" that you don't want?

If the variant of the homomorphic encryption you are using is secure, then the bad guys can not generate specifically crafted cyphertext that would decrypt into the result they specifically want.

This doesn’t really exist yet.

I would agree with this sentiment, however, there are a number of things that you can to to make the job of the attacker harder, or to notice internally when there is inappropriate usage. Service abstractions to encrypt/decrypt the data will limit the surface area where you might store keys. Logging usage to know what operations were made. Also, you can use a system like vault such that the keys are not extractable to do all the crypto operations offline thus massively increasing the chances you would detect such a breach / unauthorized access.

No, not the "we store decryption keys on our user facing app servers" part.

The decryption should happen on back-office payment processing server, with the keys held by a hardware security module.

Or better yet don't store the card details at all, but there are valid business reasons that is sometimes necessary.

In my early days, I had to write a payment processor and it pass a PCI "inspection."

TL;DR - PCI compliance was a joke in my experience.

The payment page didn't log requests (made figuring some things out hard), it submitted the encrypted card data (public key) to another server through the firewall with only a single non-standard port open, to the data store with write only access. There was then a trigger on insert that would launch a small console app that passed the decrypted (private key stored locally) card and charge information to Authorize.Net, if successful it would then write a success to the charges database, and send that charge identifier to the application's data store.

Only access to the payment server was from behind the firewall over RDP and MSSQL over non standard ports.

Just the description of this satisfied the PCI guy, and no physical inspection of the hardware was ever done. I'm not sure if what I did was best practice, and I was always so scared it would get hacked. Thankfully after so many fraud charges that my boss couldn't handle the $15 chargebacks we switched to paypal and I no longer had sleepless nights.

Can anyone recommend solutions (for those that don't know). On how to have an encrypted database for sensitive information, i.e first name, last name, ip, geo data, etc and the encryption keys not be available to hackers when they have essentially gained root?

Segmentation can help. For example, you can use envelope encryption[1] that keeps your keys on an isolated, dedicated key management server that prevents even the admin from exporting the master keys. Therefore, decryption of the data keys must be performed on the key management server.

It’s still not perfect because an attacker can potentially send requests to the key management server, but they shouldn’t be able to walk away with they keys to perform decryption outside of the system.

You then set up monitoring to watch how many decryption operations are being performed per minute or hour and alert admins when it steps outside of normal useage patterns.

It’s not perfect but can help you to 1) catch an attacker early and 2) have some type of estimate of the size of the breach when discovered. A very patient attacker may not trigger an alarm of decryption operations but in that case he’s got to work much slower, which limits the scope of the attack while they hopefully trigger something else that exposes them.

1: https://devender.me/2016/07/13/envelope-encryption/

Have a server responsible solely for decryption and audit access to it. The server doesn’t issue keys, it literally does the decryption. AWS KMS can do this for you.

The gist is you create an encryption key for your row, encrypt it using your encryption service, and store it next to your actual payload. To decrypt, ask the service to decrypt the key which you use to decrypt the payload. If your database gets popped, your decryption server hopefully didnt because you hardened it specifically.

Well, it's still tough. The question is always "gained root to what", cause at some point somebody needs a key.

Envelope encryption is a good scheme. Under envelope encryption you have a master key and per-row or per-unit keys. You store the per-unit keys encrypted with the master key, and the master key is stored offsite.

For example on an AWS environment, you would use AWS KMS with IAM authentication to handle the master key crypt for the individual keys, and the encrypted versions of the individual keys are stored in a database that you own.

You encrypt your data with the individual keys (eg, one key per user, or one key per DB row, or one key per namespace, whatever), but the individual keys are decryptable only by KMS.

Under this scheme an attacker must both be able to get ahold of the encrypted individual keys, and be able to decrypt them with the master key.

Of course this leaves you very vulnerable if the master key is acquired, but KMS does not allow you to read the master key at all, you have to use the AWS API to make encryption/decryption requests. So the vulnerability is less about how secure the key is and more about how secure your IAM setup and instances are.

The application necessarily has access to the credentials it needs to do all of this. You're only eliminating the subset of attackers who can get in, but not figure out how to imitate your use of cryptography. Is the cryptography part even relevant? Since the attacker ultimately does have use of the key material, there's no information-theoretic security provided by the encryption, only some extra work to do. Wouldn't any other indirection/obfuscation accomplish the same goal?

Or are we assuming that the decryption service enforces some policy about what and how much to decrypt?

Well, yes, if you're using AWS KMS you are able to set up policies, revoke access through IAM, use alerts and monitoring, and so on.

But you're also forgetting that most data breaches in real life involve database backups, staging servers, development environments, and people with access. So envelope encryption really helps prevent those. Even if someone gets a hand on your DB dump they're unable to use it without authorization to the KMS, which they won't have unless they can also get into your application server.

I guess I’m missing how credentials with access to KMS are any harder to come by than credentials with access to RDS, and why it’s easier to monitor KMS queries than RDS queries. Are the latter privileges more widely distributed, maybe? Where is it useful to have access to a database you can’t actually read? Do your dev/staging environments just no-op the decryption and put ciphertext strings everywhere prod shows plaintext?

Maybe your DB creds are in a secure configuration system and maybe the KMS credentials are granted by an IAM instance role. If only certain users are able to get access to config (ie RDS), and only certain machines are able to decrypt data, you now have a multilayered security approach where an attacker needs to breach both a user and a piece of hardware. If you breach a user and get your hands on a DB leak, you can't use it because you don't have access to the hardware. If you breach the instance role then you can make decrypt requests, but you would have also needed to have breached the data separately and independently.

Don't forget that data is most often leaked by people. DB backups, dev environments, inappropriate transfers. We hear about the big targeted attacks, but probably 90% of actual breaches are due to human misuse.

Depends on the use case :)

If you don't need to decrypt the data on the same system, you could always use asymmetric encrpytion, so encrypt with the "public" key and then keep the private key elsewhere.

If you require to be able to decrypt on the system in question then you need a key somewhere, so secure storage of the key is very important.

One option, often deployed in banks, is to store the key in a hardware security module (HSM) and then ensure that it's not available outside of that.

There's a load of tradeoffs to using HSMs but they can be useful where symmetric encrpytion is needed.

This was solve by banks a long time ago with hardware security modules (https://en.m.wikipedia.org/wiki/Hardware_security_module). HSMs make it impossible to extract encryption keys from the device and turn a digital security problem into a physical security problem which banks knew how to manage. Are your encryption keys secure? Yes they are right there.

Amazon and Azure offer these in the cloud.

The application has whatever credentials it needs to make decryption requests to the HSM. Wouldn't an attacker who owns it simply find them?

Typically the HSM will be part of a tokenization service that is located in a separate security zone also called a data bunker or data vault. The customer facing application will pass in the sensitive data and in its place a token will be returned. When the calling application needs the data it will pass in the token and the data is returned.

An attacker that compromised the customer facing application only has access to tokens and would be forced to access sensitive data via the data bunker. That application is heavily monitored and would hopefully be quickly noticed.

The HSM adds an additional layer of protection against loss of sensitive data via backup tapes or some insider attacks.

It's been several years since I dealt with PCI, but the app I was on the team for handled $1B+ annually. Here's just a few points that I remember.

Physically separate the key storage from the data storage. Different servers, different user auth mechanism. Make them compromise both systems.

Unique keys per client.

Use key-encrypting keys, periodically replace them.

PCI environment must be totally self contained and not mixed with standard working environment. If an employee is phished, their credentials for email/etc should not gain access to the env.

Don't allow developers to deploy or even read from prod db. Have a separate deployment team, all access to prod database audited via production tickets.

I’m not sure there is a one-size-fits-all right solution here - but having a system where decryption keys should not be extractable by software alone (e.g. an HSM, a service such as AWS KMS or the equivalents in other clouds) sounds like it would have been an improvement in this case (though with no post-mortem it’s hard to know exactly what went wrong).

Assuming a layered system, something like Vault transit encryption (perhaps with the master key in an HSM) should keep decryption keys away from front-end machines, though as ever, once you have root and can access the memory of a machine where the encryption keys are stored, most bets are off.

Something not widely used are canary tokens: https://blog.thinkst.com/p/canarytokensorg-quick-free-detect...

The general idea is to insert some kind of bogus information into a system, with no links to other systems, and then trigger a notification if someone accesses it.

One tool is to only get access to this data from the Web FE via an API. This API is then monitored for spikes in usage, so if a given FE is suddenly requesting a lot more than normal it gets flagged. It doesn't stop targeted attacks, but does show the big ones - another reason why security needs to be multi layered.

Keep in mind they've been compromised for 4 years. For that time duration it's not just about protecting data at rest, it's also about protecting data as it is processed inside your systems.

>hackers when they have essentially gained root?

You are already fucked at this point. I would focus my attention on prevent hackers from 1) gaining any unauthorized access. 2) from doing any sort of privilege escalation from a restricted account/service.

Maybe keep the keys only in memory in each application? Pull them from some very well monitored service once at startup? Anyone who gained access to the server would have to check the program's memory to get them.

How is that the "Oh dear."? Do you work for a bank?

That seems like the least interesting bit of information here for individuals.

It's really bizarre to me that people here seem to consider their credit card numbers more sensitive than their travel history or even contact information.

Why is it bizarre? Most people wouldn't consider their comings-and-goings to be worthy of any privacy protections, though some people have non-criminal reasons to want to protect that privacy. Contact information likewise is generally public information, some people may have non-publicized means of contacting them though that they want to keep private.

Generally people's level of care will correlate to what nefarious purposes the data can be used for. There aren't that many such purposes with the data exposed here until you consider the passport number (probably something more secure than a SSN for most Americans?), payment info, or login info. The purposes I can think of for the other data are reliant on the metadata that so-and-so was at that location at that time, when I might have believed something else.

well possibly not. If the attackers got the keys to decrypt payment details that means that victims, in addition to the identity theft risks, may also face fraudulent transactions on those cards as a risk.

Even if the payment information is in plaintext it'll primarily be the banks who are affected, not the customers.

I'd be far more concerned about my privacy than my banks money.

Anyone hit with fraudulent charges has to a) notice them, b) dispute them and c) go through the annoying process of having your card re-issued.

As to privacy, after the number of breaches of personal info. there have been, I'd be inclined to let the idea that this level of personal data is private :)

>a) notice them

Don't you review your charges regularly? Even without fraud you're bound to be losing money to erroneous charges if you don't.

>b) dispute them

I've usually just had to submit a quick online form consisting almost entirely of checkboxes, while yeah that's some work it's still not going to take more than a minute of my time.

>c) go through the annoying process of having your card re-issued.

Seems like this is vastly more annoying with some card issuers than others.

I notice sure, but that doesn't mean all of the x million people who've just had their cards stolen will.

Also not all of the cardholders will be online customers, so they'll have the delights of call center processes to go through.

Whilst it's obvious you don't regard loss of your credit card information as a serious inconvenience, I don't think that's necessarily a universal sentiment.

>Don't you review your charges regularly?

I work at a bank, and you would be surprised how many people don't know how much they have in the bank until they get the message saying there's no money.

so many people do this. As long as they don't get that message, they don't worry about how much is in there.

People who use debit cards like credit cards can be seriously inconvenienced by having their checking account drained.

and according to the article: "It said an internal investigation found an attacker had been able to access the Starwood network since 2014." 4 years of access, surely the attacker didn't manage to get hold of these keys

Curious if it matters much that the keys were stolen too. How long to brute force a 16 digit number? With a known subset of 4 digit leaders, and being able to cull the list first for ones with good luhn checksums.

Edit: Ugh. Yes. My mind was apparently elsewhere. Keyspace would drive the time needed.

Though it is a relatively short list of known plaintext. especially if you focus on the bin ranges for say the three most popular banks in the US.

But, only interesting if Marriot was using some encryption with a small keyspace.

Are you really suggesting that bruting cc #s would be a viable method of committing fraud?

If yes, how much would you expect the bruteforcer to earn per successful attempt?

To construct a valid card identity you need more than just a 16-digit number that passes the luhn check.

At the very least you'll need cardholder name and expiry to match up, and hopefully you'll need CVV/CVC as well.

Of course I really hope the Marriott weren't storing CVV in a reversibly encrypted format.

> Of course I really hope the Marriott weren't storing CVV in a reversibly encrypted format.

You're not allowed to store the CV2 in any form that could be recovered (i.e. plain text or reversibly encrypted) or brute forced (i.e. hashed/salted). PCI rules say you simply aren't allowed to store the CV2 after the call for an authorisation, as it's no longer required. If they were storing the CV2 then they're in trouble.

The linked article reads like most of that was not encrypted. The CVV wasn't listed as being stored, but CC number, name, and expiry, without the CVV are usable in the US, even online in many cases. A CC charge without CVV doesn't hard fail, so it's the merchants choice as to whether to even ask for it.

IIRC (although I could be wrong, been a while since I was in banking) that not asking for CVV changes the balance of who's liable for fraud?

But you can buy that information for $1 a pop (or far less if you buy in bulk, think $0.3 or so).

Credit card fraud is far more involved than just getting payment information, you won't succeed at ordering anything of value without understanding how anti-fraud systems work.

sure it is, it's still more valuable than basic personal information.

I'm guessing that higher end criminal spend quite a bit of time working out how to bypass anti-fraud systems, it's an ongoing battle.

Actually using the card information is almost entirely left to the lower-end criminals, it's just ridiculously difficult to scale.

After spending years hanging around in those circles I'm rather convinced that the only people making real money with credit card fraud are the shops, hackers stealing the cards and reshipping services.

The biggest buyers on the shops seemed to be criminal gangs engaging in relatively small-scale fraud maybe moving hundreds of thousands a month.

I don’t follow. Why is it the size or structure of what’s being encrypted that matters for brute force time rather than the key space of the encryption algorithm.

Password hashing is different because there is no key space.

Card information has no value because it is very easy to reissue a card. Things like name or phone number, or address, cannot be changed so easily and have much higher value.

Things like name, phone number, and address are not secret. They are published in the phone book.

Card information is literally cash money. It's volatile and subject to being rendered defunct, but it still has a lot of value if you can buy a bunch of prepaid gift cards with them before it's deactivated.

Is payment information not typically salted & hashed like a password?

Imagine you could make payments with a salted and hashed credit card number.

Except that doesn't really make anything better, because now an attacker could simply use that salted and hashed credit card number elsewhere to make payments too!

The real solution is to use something like OAuth for payments. You authorize a merchant to take ongoing payments from you, and the card issuer gives the merchant a token which is only useful for making payments from you to them, and can't be used to make payments to anyone else.

Isn't that basically what Apple Pay does?

Apple pay uses a card number per merchant, but that card number could still be used at any other merchant. Eventually that loophole could be closed, but payment systems move like molasses... Expect it to take 20+ years...

It simply means if the card number is used for fraud, it'll be easier to track down who leaked the card numbers.

Whilst you can't hash it like a password as you need to be able to transfer it onwards, modern systems make tokenisation possible, where you immediately pass the details to the card processor, don't store them at all, and get back a token that can be used for that transaction.

https://squareup.com/gb/townsquare/what-does-tokenization-ac...

Not if you need to reverse it to actually send to payment card processor.