ircmaxell makes a good point in that comment -- not that the concept of a pepper doesn't work, but that a two-way encryption function is a better choice than a hash function for applying the pepper.

Your pepper will be a long, random key that is known to your app server but not your database server. If you store passwords as:

     bcrypt(bcrypt(password, salt), pepper)

then you spend a lot of cycles bcrypting your long, random key, which is pointless (it should already be long enough to be un-brute-forceable), and you lose the ability to rotate keys or ever stop using this scheme in the future. There's also (in theory) some risk that nested bcrypt() doesn't work predictably.

If you store passwords instead as:

    encrypt(bcrypt(password, salt), pepper)

then you can routinely rotate peppers through a simple one-time decrypt-and-encrypt step on your app server, instead of endlessly nesting peppers as suggested in the filippo.io blog post.

The rest of it is more of a qualitative question -- what's the risk that someone gains access to your DB but not your app server, vs. the risk that in implementing the pepper, you somehow screw up and store something easily crackable?

Once you use Encryption, it's no longer pepper, it's encryption with weak key storage (hard-coded). At that point, genuinely, why not just follow best practices and store the key securely (e.g. using an HSM)?

Certainly storing your key with an HSM is superior and protects against a slightly large class of attacks, but in terms of getting the same protection as salt+pepper was designed for (limiting the damage of a DB-only compromise), encryption, even with weak key storage, is superior. Adding an HSM is just ways we can improve that even further (which aren't really even feasible with the salt+pepper idea).

Because nobody knows how to do that and it is likely extremely expensive by every metric.

And, no, I am not being facetious by saying nobody knows how to do that. I am being quite literal. Have you ever done that? Do you know how? Do you even know what you would google to figure out how?

I'm yet to see my favorite library of course's documentation on a HSM. How do you do that in e.g. PHP with MySQL? MVC with MS SQL? Java with Oracle?

It's worth it. And yes, I have done that, it's not a mystery, they all provide a API/ABI for interacting with them[1], such as OpenSSL's Engine API or PCKS#11.

Edit: There's even a project to literally do this directly from PHP[3]

[1] http://stackoverflow.com/questions/10796485/interfacing-with...

[2] http://en.wikipedia.org/wiki/PKCS_11

[3] http://stackoverflow.com/questions/3231293/how-to-interface-...

Eh, I had a Luna CA3 sitting on my desk for a while. HSMs aren't that exotic. I'd be more concerned about the HA aspects which could require extra code. I suppose you could just shove a USB HSM in several of your servers and encrypt every password with at least 2 of them.

Christ you weren't kidding about it being expensive. Amazons Cloud HSM is $5k upfront and $1.8/h thereafter, which is completely infeasible for the agencies I've worked at. A shame, I'd love to do the right thing there :(

That's why AWS also started offering Key Management Service: https://aws.amazon.com/kms/

You don't get your own HSM but it's MUCH cheaper ($1/key/month) and more scalable and available than an HSM.

That's brilliant. I'm definitely going to use that, thanks for posting it!

I think AWS provides both key management services as well as cloud hsm.

I have not used either myself, but I would imagine the documentation is quite good.

I've done it using a $100 Cryptostick (Now Nitrokey). Here's a guide from Mozilla: https://blog.mozilla.org/security/2013/02/13/using-cryptosti...

https://www.nitrokey.com/

Joining the chorus; I've done it. Expensive on some levels, but very cheap on others. Dealing with keys as physical objects is a lot less stressful.

> what's the risk that someone gains access to your DB but not your app server, vs. the risk that in implementing the pepper, you somehow screw up and store something easily crackable?

The question is irrelevant anyway, because if someone gets access to your app server, they get the secret in both cases (pepper and encryption). So the first risk there is present in both cases, which just makes this a static "what's the risk that you screw up in implementing the pepper?"

The question is not irrelevant. It's blindingly obvious that salt+pepper protects against DB-only attacks. Injection that does a DB dump, or vulnerabilities in the DB server that don't exist in the app.

So yes, S+P protection won't save you if the app server is completely compromised, but it does protect you in case DB only is.

And the ability to S+P seems pretty simple to implement and document. Why is everyone panicking that it's hard to do?

I've already seen one person in this thread taking "S+P" literally, and suggesting `bcrypt(cost, salt + pepper, password)`, which has a very obvious and critical problem. Other "peppering" schemes in the comments here also have significant weaknesses. So yes, even a simple idea like this is easy to botch in implementation -- even if the implementation is just a one-line change!

What is the problem? Does bcrypt truncate "salt+pepper"?

Because the salt is in the hash!

$2a$12$GhvMmNVjRW29ulnudl.LbuAnUtN/LRfe1JsBm1Xu6LE3059z5Tr8m

$2a means it's using bcrypt version 2a

$12 means 12 rounds

GhvMmNVjRW29ulnudl.Lbu is the salt

AnUtN/LRfe1JsBm1Xu6LE3059z5Tr8m is the checksum

I read your question as comparing the originally-proposed pepper approach vs the encryption approach, and I said the question was irrelevant because both approaches have the identical risk that the app server is compromised, but only the originally-proposed pepper approach has the risk that it's not actually secure (e.g. because naïvely composing hash operations may be bad, and adding static bits to passwords may also be bad).

But upon reflection I believe you were instead using the term "pepper" to include the encryption approach as well and merely trying to question whether the added security of requiring an app server compromise is worth the risk that you still screw it up somehow. And to that I'd say that it's not difficult to apply an existing block cipher algorithm when storing/retrieving password hashes so I think the risk there is low.

> Why is everyone panicking that it's hard to do?

Everybody's panicking because the originally proposed pepper implementation is a really bad idea. That approach has not been researched for security implications, and there are many reasons to believe that composing hash operations without using a specially-defined operation like HMAC is bad, and adding static bits to the salt or password (e.g. if you use 'pepper.salt' for the scrypt salt or 'pepper.password' for the password) is also bad.

However, I believe the approach of using a block cipher to encrypt your hashes with an app-wide password is reasonable. It's not composing operations badly (encrypting a 256-bit string, or whatever scrypt emits, is perfectly reasonable given a secure key) or otherwise providing an attack vector on the hash algorithm.

The biggest risk I can see with this approach is you have to make sure the pepper is stored securely on your app server, never visible to the database server (and never accidentally committed to an open-source repo, if you're using an open-source server). Not just that, but you have to make sure that you don't accidentally lose it either (or you'll have instantly lost all your accounts). But this is a solvable problem.

Because it's harder to implement and easier to mess up than using standard encryption, and has unknown characteristics due to the fact that it's rolling your own crypto. Why would you spend the time and effort to implement it when you can do something that is objectively better in every way?

You never know. Some developers might not be smart enough to use an SSL connection from their app server to their DB server. The influx of newb devs out there is growing and security is usually not high on the learning list or the new devs priority list. If so, hackers could possibly get access to the DB without even needing the app server in the first place. Get your certs folks!

I don't see how this is relevant. There is no reason to believe that the hacker needs to get at your app server in order to access your DB server in the first place.

That sounds like fuzzy scare-mongering to me.

1) You should not invent your own algorithm. That's a given. That's why you use bcrypt/scrypt.

2) It's not abusing the algorithm, it's using a longer salt (in the concatenation case).

3) There's nothing wrong with nesting algorithms (just remember to use hex/base64 encodings, not binary). For example Facebook passes passwords through half a dozen algorithms. They call it the "onion". And it includes a pepper.

4) As for being effective, I think the SQL injection case speaks for itself.

5) As for rotation - just don't do it. You pepper gets compromised? Who cares, add a new one on top of the old one.

Also, I'm confused at how the proposed alternative would be harder to get wrong:

> Encrypt The Output Hash Prior To Storage

Correct me if I'm wrong, but an algorithm is a set of steps, which this looks like to me

    salt = urandom(16)  
    pepper = "oFMLjbFr2Bb3XR)aKKst@kBF}tHD9q"  
      # or,        getenv('PEPPER')  
    hashed_password = scrypt(password, salt + pepper)  
    store(hashed_password, salt)

That is an algorithm, which composes bcrypt with pepper.

The idea of not using key-rotation alone is insane, but lets just focus on your last point

    Also, I'm confused at how the proposed alternative would be harder to get wrong

Really? AES literally has hardware support, and can be done in a single call, and has been studied for years. How can that reasonably be considered "harder" to get wrong than something proposed by some random guy on the interwebs?

Outside of Peer-Review, what reason would anyone have to use the pepper scheme? As others have posted, there are several community members who's opinions do matter due to extensive research and body of work

Your second point might be dangerous - your salt values are no longer random but heavily biased and knowing that all salt values share some common bits might provide a new attack vector.

Is this true? I would think that static bits are no more dangerous than not having the bits at all.

Here is for example an attack recovering a 384 bit ECDSA key [1] by knowing the five least significant bits of the nonce (obtained by a side channel attack) for 4000 signatures. Now hashes and signatures are obviously very different things but I would not bet on the fact that a bias in the salt does not matter.

[1] https://eprint.iacr.org/2013/346.pdf

ECDSA is a completely different beast. I'm not aware of a modern password hashing function that would be broken if it was given a non-uniformly random salt. If such function were to be submitted to PHC (https://password-hashing.net/), I would consider it to be a disqualifying factor.

For password hashing purposes, salt doesn't need to be uniformly random, the only requirement for salt is to be unique and unpredictable to the attacker (see http://crypto.stackexchange.com/questions/6119/hashing-passw...). Most password hashing functions use a cryptographic hash on salt.

This particular function, scrypt, uses one-round PBKDF2-HMAC-SHA256 to mix password and salt:

https://github.com/golang/crypto/blob/master/scrypt/scrypt.g...

PBKDF2 feeds salt, basically, into SHA256:

https://github.com/golang/crypto/blob/master/pbkdf2/pbkdf2.g...

If there was a known attack it would obviously be a non-starter. But why take the risk that someone will come up with something comparable - in the broadest sense - to say the differential attack on MD5 that allows exploiting known bits in the salt if there is no need to?

Interesting. Thanks for the link!

> 2) It's not abusing the algorithm, it's using a longer salt (in the concatenation case).

But PBKDFs like bcrypt and scrypt are not designed to keep the salt parameter secret; in fact they assume the attacker knows the salt. And so if they happen to reveal the salt to the attacker, this is not considered a bug in the algorithm and won't have been flagged or fixed by cryptographers.

(And more importantly in practice, the implementations of these algorithms aren't designed to keep the salts secret.)

The "concatenation case" completely defeats the purpose of using a pepper and leads me to believe that you're not qualified to be giving this kind of advice.

In what way it defeats the purpose of using a pepper?

Why combine hashes? Just use XOR. If bcrypt can protect "not_my_password", it can certainly protect "KKQ{H]zTDWVSJVA", which is the former XOR'd by "%$". XOR doesn't decrease the keyspace (or change it in any interesting way), so any attack on XOR is an attack on bcrypt; XOR is fast enough to evade any sort of timing attacks, too.

Careful. If an attacker is ever able to observe the output for a very short password - one byte, say - then only one byte of your secret salt is used, and the attacker could begin brute-forcing the secret salt starting with the first byte.

XOR could also result in NULL bytes anywhere in the hash input, which could drastically weaken passwords,. For example, bcrypt ignores any password characters after the first NULL byte. This is especially bad if the attacker can supply their own passwords, doubly so if they can observe the output, since they can then easily brute-force individual bytes of the secret and use that knowledge to intentionally create NULL bytes in the hash input.

> XOR doesn't decrease the keyspace (or change it in any interesting way), so any attack on XOR is an attack on bcrypt

I wouldn't make any statement like this unless you've actually gone through the steps to prove it.

>Careful. If an attacker is ever able to observe the output for a very short password - one byte, say - then only one byte of your secret salt is used, and the attacker could begin brute-forcing the secret salt starting with the first byte.

Can you be kind enough to explain with a very simple example?

I would appreciate understanding your point - Thank you

Using the scheme above, a password of "a" would result in a hash H('a' ^ x) where x is the first byte of the secret salt. The attacker can simply test all possible values of x (there are only 256) to determine the value of that byte. Knowing the first byte, the attacker can then look at a two-byte password and repeat the process to brute-force the second byte.

(It's also possible to brute-force more than one byte at a time; it would just take longer. For example if the shortest password the attacker can observe is 6 bytes then they would need to try 2^48 possibilities.)

That's a rather comprehensive answer from @ircmaxell over there, hm. Interesting, thanks for sharing.

so the downsides are "it's not maintanable" and "don't roll your own crypto". I think they are negligible compared to the upsides.

Not at all. "don't roll your own crypto" is a downside that can lead to things completely falling apart or weakening the system overall.

The real downside is that there's a better, proven way to do the same effective thing, which is make a database-only compromise require additional work, without rolling your own crypto. It also supports doing things retroactively for real (not some of the hacks being discussed in this thread) and key-rotation. All the upsides, with none of the downsides.

> "don't roll your own crypto". I think they are negligible

Please do not ever consider "rolling your own crypto" a walk in the park. Unless you have a serious security background and some actual cryptography education and research never, never, NEVER do this. It is not negligible, and it is not safe.

The fact that the pepper can't be changed/rotated far outweighs any upsides

You can change your pepper by double-peppering your existing password database:

  scrypt(scrypt(scrypt(scrypt(password, salt), pepper2013), pepper2014), pepper2015)

https://blog.filippo.io/salt-and-pepper/

Sounds like a maintenance nightmare. Need to ensure you keep your tongue straight when partitioning or restoring databases, migrating/splitting to new apps etc.

You consider "not maintainable" to be an acceptable downside to _anything_ you're doing with your software? What?

I wouldn't say negligible, but rather not slam-dunk convincing