A pepper is essentially a secret encryption key (it's a long secret string that's added to the password and the salt to ensure more entropy). With cloud key management services (e.g. both AWS and GCP have a KMS), I think it's more beneficial to just encrypt the hash before putting it in your database. Process looks like this:

Upon password creation:

1. Generate hash as hash of password + salt.

2. Encrypt the hash with a public key from KMS (you can store the public key in your server code).

3. In your database store the encrypted hash, the salt, plus some "key ID" that identifies which KMS public key you used (this is so you can rotate keys later).

Upon user login to verify the password:

1. Retrieve the user's encrypted password hash, salt and KMS key ID from the database.

2. Make a call to KMS to decrypt the hash (KMS internally stores the corresponding private key but never lets you access it).

3. Then hash the password the user entered + salt and compare it to the decrypted hash to see if there is a match.

Benefits of this are:

1. If an attacker steals your database, they can't decrypt any of the passwords or the password hashes.

2. KMS never exposes the private key of the async key pair, so you know this won't get exposed either. The only way to decrypt something is to make an API call to KMS.

3. Thus, the only valid attack really is if the attacker is able to gain the same access privileges as your server. But even then they still need to call KMS one-at-a-time to decrypt hashes, and all of those KMS calls are logged in an audit trail, so it should be much easier to see if you have anomalous calls to KMS. There is a huge benefit here in that it is impossible to do bulk decryption without a giant audit trail.

We do something similar for storing all DB entries (since our data is sensitive, as we're a financial services company). Even if someone gets access to our DB, all they'll get is garbage :)

Bookmarking this. Additionally I assume you could also limit access to KMS so that only your servers can issue requests.

Yep. Not sure the details of AWS, but in GCP access to KMS APIs and specific keys is controlled by IAM, and you can set "conditions" on IAM policies to restrict access by things like IP of the request: https://cloud.google.com/iam/docs/conditions-overview

KMS is managed by IAM so you can attach roles or use STS (create an access key / password) that can assume a role that can encrypt.

pepper instead of salt is a bad idea because if your pepper is leaked the attacker can brute force all of your passwords at once.

The main argument I've heard against pepper is because people are afraid of losing the pepper. It either needs to be directly in your code which is easier to leak, or out of band which is easier to lose.

edit: Does anyone else want to goto waffle house whenever they talk about salting and peppering their hash? Too bad the closest one is an hour away.

If the pepper is stored as an environment variable, adding it in addition to the salt can be a minor increase in the password security. The thing is it often isn't that hard to get if someone has access to your server.

If it's just the db that is exposed though, it could be a small added layer of protection.

And yes, I would not ever do it instead of a salt.

Would in this scenario be running each service (database/backend/frontend) in separated clouds/envirorments be benefitial to mitigating risks?

Let me explain. for example, I might have my NextJS frontend in Vercel using it's secret management /env tools.

The backend as a vanilla node-apollo-server-express could probably be on a cheap VPS, being monitored/restarted/load-balanced by PM2.

The database would be cloud, either PostgreSQL as a Service, or Fauna or something.

Would this scenario be better than just cramming everything into a VPS and trying to get that as secure/closed down as possible and be done with it (do monthly updates and whatnot)...

I've faced recently this conundrum at job. Small new app will not have more than a few hundred concurrent users optimistically...

As @wongarsu said, I think running the DB separately makes a lot of sense. Also, that (in theory) protects any ENV variables on the main host server from exposure if your DB is compromised. And the DB is likely better protected and up-to-date if you use a DB-as-a-service. But I wouldn't go extreme either. There are a lot of hidden costs associated with utilizing additional machines, whether they be via lamda functions, VPSs, etc.

One other good thing about keeping things separate is updates in theory should be easier, as besides OS stuff, you only have a handful of applications on each machine. But again, for small apps, it's generally not worth the extra complexity. 3 platforms is likely as far as I go (client-side server, API, DB) in the majority of cases with small apps, and most of the time I'd probably just go with 2 assuming Node can serve both the client and API.

Theoretically, more separation means more security. On the other hand, securing many things is harder than securing few things.

PostgreSQL as a service (or whatever DB you prefer) is worth it, that means somebody else can get backups, security relevant settings, patches, version upgrades etc right. Everything else is probably fine on one server, but it doesn't sound like that would be much besides the backend in your case anyways.

> pepper instead of salt is a bad idea because if your pepper is leaked the attacker can brute force all of your passwords at once.

This isn't true, you could simply do encrypted_pw = md5(pepper + md5(password)) or whatever.

Edit: Getting a bunch of comments on this, just want to clarify that I used md5 purely to illustrate a one-way hashing function. In actual practice, you'd use something else (HMAC with SHA256 most likely).

> md5(pepper + md5(password)) or whatever.

(Ignore md5)

No, I strongly recommend not to do so. The reason you put an salt in is to prevent multiple hashs to be the same (because people use the same password) with a pepper you still protect against cross reverencing the hash with other databases, but any two users in your database will still have the same hash and people tend to reuse passwords (or similar passwords) so this is a very real attack vector to get passwords without really braking any hash fully (E.G.: user with known password on different platform => try out similar passwords => broke hash for anyone with similar password).

So a unique per hash specific salt is the most important thing to do.

Pepper/shared secret can make it additionally harder to crack any hashes as while you know all salts (they are stored alongside the hash) you don't know the pepper.

Lastly there is additional data (AD) (named sometimes differently). Which can prevent some form of hash reuse attacks where you e.g. find some form of attack which allows you to override hashes+salt in the db (but not more). Then you could rewrite all hashes to known ones and get access. Tbh for many systems if an attacker can do something like this they don't need to do that anymore. But for other (often large and complex) systems it's helpful.

The idea behind AD is that you (somehow depending on algorithm) include some additional data which needs to match. The most common example is to use the user id (if immutable) as AD so this hash+salt(+pepper) is only usable for given user and never for any other user.

If you ever write a auth sub-system for a big enterprise system I would recommend you to use salt+pepper+AD(uid), for everything else I would think salt+pepper is enough. But never should you use a hash without unique salt under any circumstance. It's always the wrong path to take.(For password hashing.)

Or at least that's my opinion.

Is it worth it to salt in scenarios when the passwords (actually tokens) themselves are guaranteed to be unique and instead only use pepper?

Short answer: if they are unique because they're a small sample from a large space (e.g. UUID v4) no salt is needed. If they're unique but maybe predictable, salt.

Thanks! Yes, they are random GUID-like values and are short lived (2-4hrs). Had a need to store them for a reason and decided to only add pepper to hash considering they are unique and short lived anyway.

SHA256 is also no good for storing passwords, you need to use a PBKDF like scrypt, bcrypt, or pbkdf2.

The SHA-family cryptographic hash functions are purposefully designed for throughput, if you combine them thousands of times like in PBKDF2 they can be fine. One round of SHA256 is trivial to brute-force especially with the plethora of ASICs available.

HMAC is also completely unnecessary here, and see the article title for your variable naming: it's not encrypted_pw it's hashed_pw.

I know this is just a comment on HN, but I want to point out that MD5 is not a password hashing function, and is broken even for a number of other hash purposes. For passwords something like argon2id or another modern slow hashing function is appropriate, and for general purpose, SHA2 and SHA3, as well as BLAKE2 (and maybe BLAKE3) would be good choices.

Further, there’s so little reason not to salt, so you ought to do it. It’s built in to the hash strings in most modern pw hashes. Peppers are also a great idea. Defence in depth!

MD5 is broken, but HMAC-MD5 is still impossible to break afaik. And md5(pepper + md5(pw)) is basically HMAC.

Not commenting on the overall idea, just pointing out that the criticism of md5 happens not to be valid in this case. I was surprised too.

The discussion of "MD5 collision resistance is bad, but md5 preimage resistance is still ok, so using md5 for applications where preimage resistance is required is ok" is I think completely valid (however pointless it is)

However for password hash functions to be useful they also need to have a high work factor to make brute forcing even more difficult.

The only unknown in that equation, once you get hold of pepper, is password. So, what added security does it provide, other than requiring another md5 computation?

How does that help? They can iterate through possible passwords and generate md5(pepper + md5(password)) just as easily as md5(pepper+password). The point is that they can iterate ONCE and match against all passwords in the DB. With salt they have to iterate for each row in the DB which is much more time consuming.

If that gets leaked the attacker could brute force that, and match the results to your entire list of hashes. If they're salted, they would still have to brute force each password individually.

Totally agree with this, hence the caveat that the code doesn't get leaked/compromised :)

I think you don't quite understand the purpose of a salt. Pepper and salt protect against very different things. And frankly, salt is much more important.

am I missing something here? the attacker still only needs to know the pepper to brute force all the passwords with this scheme. Since the pepper is deterministic, and especially with md5 which is already extraordinarily quick to reverse, the attacker can just take md5(pepper) then do the extremely quick operation of the hash extension md5(password)

Noone uses MD5 in the real world. SHA-256 is generally the minimum now (for anyone serious about passwords)

Please not md5. Please.

> The only caveat is that your database isn't coupled tightly with your application code, so your pepper remains secret even if your DB is breached (which is usually the case).

There are many attack scenarios in which storing a secret separate from the DB doesn't get you much at all. Suppose an attacker finds an RCE vulnerability in the application – then they can slurp up the contents of the DB, but they can read the pepper from the configuration too.

Suppose they just have a SQL injection – they can't directly get the pepper from a SQL injection, assuming it is being stored somewhere outside the database. But, it may help them do it indirectly – for example, they could UPDATE their account to have admin privilege, and then access admin-only features of the application which may end up revealing the pepper. I've seen many apps which have admin-only screens to run arbitrary scripts, view configuration files or environment variables, install plugins, etc – those kind of features are helpful in supporting the application, but can be used to turn a SQL injection into more complete control over it

> Suppose they just have a SQL injection ....

But what about having separate auth sub-service with separate database.

Who says a SQL injection allows yourself to update your account to be admin. (SQL has their own permission system even through its not used that much).

Who says making yourself application admin allows yourself to do more then banning users.

Sure for many especially smaller systems pepper won't help much and AD (additional data) is even less likely to help.

But for many (especially bigger/enterprise) systems this might very well not be the case.

Furthermore pepper usage tend to not really hurt.

I think it's a fallacy to assume just because many less well build systems allow SQL injection to escalate that all do so and pepper is useless. Especially given that most systems need to have some config secret management system anyway and as such adding a pepper tend to be somewhat cheap to to (dev effort) and doesn't really cost much at runtime either (normally).

> But what about having separate auth sub-service with separate database.

Sure, my point was mainly about classic monolithic apps. If you split your app up into lots of separate micro-services, my point may no longer hold.

> Who says a SQL injection allows yourself to update your account to be admin.

I wasn't talking about a database superuser account, I was talking about an application-level admin. Usually there is some database table which stores user permissions, and an attacker can do an UPDATE/INSERT on that table to grant an ordinary user account full admin permissions, or even create a brand new user with those permissions. All this can happen within the single ordinary DB account used by the application (given most applications only use one DB account)

> There are many attack scenarios in which storing a secret separate from the DB doesn't get you much at all. Suppose an attacker finds an RCE vulnerability in the application – then they can slurp up the contents of the DB, but they can read the pepper from the configuration too.

If the auth is a service accessing only via some remote access protocol that only allows a minimum number of operations that do not map directly into database commands they cannot exploit RCE in the app to get auth database.

Such service can also enforce reasonable rate limits and quotas, something that a regular direct database access can't enforce.

It's really hard to store pepper in a system meaningfully more secure than the password hashes themselves. For any suggestion of a safer way to store pepper my response is "store the password hashes there instead". Until you get to the point of decrypting password hashes in a TPM I don't see any benefit and at that point you've switched to a hardware solution.

That's not the argument though - the argument is that storing high security rarely accessed information ( hashes of passwords / encrypted passwords ) that are needed for a very limited number of well defined operations on the same system within the same security domain with the low security often accessed information with a high number of not well defined operations is what responsible for majority of hashed/encrypted credential leaks anyway.

You are going to have a much harder time getting data from a service that only speaks HTTPs with 6 endpoints that can only be fed 1 type of JSON with only specific fields present per end point rate limited to rates per second that make sense for those individual end points with a database backing it being internal to that service which gets 15 deploys per year than from a Gigantic Database Supporting Application That Does Everything For Users and Business and Marketing and Analytics that keeps changing based on weekly sprints.

Admin tools should probably be IP whitelisted instead than open to the world if admins are few?

Then the attacker will just add their ip to the whitelist?

Is that not the default? Err on the side of better security?

Yes, furthermore (db) admin login should preferable be done over a different SQL user so that a normal usages SQL injection can never be used to update anyone to have any admin rights.

But then just giving the default user admin rights can make many (not so good but widely used) deployment tools and methodologies and similar much easier to use so you see it quite often.

I have seen it a few times. Default db user being a db admin because the ad-hoc written deployment script during initial development was never updated and security didn't matter back during per-release bootstrapping. ;=(

> Yes, furthermore (db) admin login should preferable be done over a different SQL user so that a normal usages SQL injection can never be used to update anyone to have any admin rights.

You have to distinguish application admin rights from DB admin rights. Most apps run with an ordinary DB user account, and even with a SQL injection bug in the app, you'd need to find an additional database vulnerability to upgrade the ordinary DB user account to a DB admin account.

However, for most apps, what is really valuable is the data, not the database, and for the data you don't need the DB admin account. The app's own users are often stored in a database table, with some flag (or something more complex like a separate group membership table) used to give an app user admin rights in the application. The point is, those admin rights can potentially unlock maintenance-focused features which could be used to extract the password pepper from the configuration. (Of course, as other commenters have pointed out, this is much harder in a microservices architecture with a user authentication service than in a classic monolithic app.) Giving a user admin access via SQL injection can also have other benefits – while data theft can be done via a SQL injection vulnerability, it can be cumbersome; using REST APIs, file export screens, etc, can potentially make data theft quicker and easier. The cost is increased risk of discovery, since auditing of admin rights may discover some unexpected new user granted admin rights, or some existing user having them unexpectedly – by contrast, data theft via a pure SQL injection with no data changes made would not be detected by an access audit.

I have wondered the same thing. Make it a piece of data that comes in from the environment or a secrets manager. It’s not bulletproof, but if done correctly seems like it makes the whole hashing scheme a little better.

If you are doing this isn't it pretty much the same as encrypting it with a key?

encryption is non-deterministic

Some encryption algorithms are (effectively, but not literally) non-deterministic, e.g.: DUKPT, RSA-OAEP

Most encryption algorithms are deterministic, and most password authentication schemes depend on that fact.

Passwords aren't encrypted, so why would deterministic encryption matter?

Err, yes, good point. And if I had read the article first, I would have had the proper context of technical specificity loaded before commenting.

Nevertheless, most encryption algorithms are deterministic. :)

if that was true, this article wouldn't exist: https://en.wikipedia.org/wiki/Deterministic_encryption

you really shouldn't be implementing your password hashing algorithm from basic crypto primitives. s/bcrypt works right out of the box with no config.

> It's arguably much easier to implement than salt

Salting is easier since it can be completely wrapped up in the key derivation function. The programmer doesn't have to know anything about it, they just use generate_key(passwd) and check_password(passwd, key).