Can't you encrypt the entered email and search with that?

Password hashes are handled separately using bcrypt - another story.

For the User.email field, we crypt/decrypt in the ORM/DB layer. So we search with the encrypted value that is stored in the db. App code provides plain-text email, ORM encrypts and searches with that value. Symmetric encryption of short strings is pretty fast and this is not a high volume API.

We use rate limiting on the API to protect against DOS on this API.

If it's symmetric, then doesn't an app vulnerability make it possible for you to leak your key? Then the app also contains access credentials to the DB, so vulnerabilities in the app will still lead to real access to the DB, right?

Yes, we're not protecting against the app being attacked, the key being jacked and then the attacker getting access to the db. Rather, we're protecting the database against being accessed directly. That could happen due to an error in configuring network access to the database or another service could be compromised that has access to the same database -- as has happened to many mongo databases over this year.

So your solution to fixing a basic ops problem 'software package A was exposed to the internet with(out|effectively no) auth' is: encrypt shit in an ORM layer and destroy your ability to do anything beyond absolute string comparison queries.

No thanks.

No, we don't encrypt indiscriminately. We selectively pick fields to encrypt - fields that are highly sensitive.

And you are right, we do this because an ops error can easily make a mistake sometime in the future and probably will one day. We all make mistakes and defense in depth is all about that.

It does seem like emails are a good fit for encryption. I can't see wanting to do anything more than simple equality checks.

I could see a use case to see which emails are from what TLD.

For instance, all @gmail, all @yahoo, all @aol.

Maybe you want to do the cool "hey we noticed your email on haveibeenpwnd, you should change you password here just in case". In which case, anything other than plain text could prevent that from happening.

Hashing an email in that sense gets much more difficult, no?

If you have enough emails that you can't SELECT * FROM users and do that query in memory, you're probably in a spot where you should not be picking security advice from a blog like this.

(That's not a negative as to this blog, it's really good and I've recommended it to multiple clients already, but that level of acumen should already be assumed at that scale. If you don't have it, this blog post is insufficient.)

This is an interesting solution, thanks for sharing. This would be a cool feature to build into open source ORMs.

This is a good idea, but only for fields which can never be used in range based index lookups.

I see. Makes sense. Thanks for sharing this approach.