It's just such an easy mistake to make. Some new ops guys logs all the traffic through a subsystem and boom - plaintext passwords.

Yeps. And at Facebook-scale, when you realize the issue after it has went live you have already "stored hundreds of millions of user passwords"

That's a good point in favor of less hacking things together and more engineering - Facebooks seems to have not embraced strict engineering processes which are the only way to tackle these things. Facebook has to change it's own principles to succeed long term.

I would say that this needs to change on a broader scale within the software development community than just with Facebook.

An easy mistake with serious potential consequences. I'd be interested in hearing how Facebook will prevent this from happening again in the future

A lot of folks are angry about this, but I'm not seeing much in they way of "this is how that should be handled" aside from "be vigilant." The closest is "encrypt logs," but those logs exist to be seen, else they wouldn't exist.

Hash client side. Send hash.

This is how it is done for 20 years (by those who truly care about the users).

If this is so common, it should be easy for you to provide a link to the javascript password hasher on a site you use. Does this site do that? Pretty sure it doesn't. But you sound very confident about this, so I'm sure you'll have no trouble finding some other site that handles its passwords this way.

What?! No!!!

As many have said before, transmitting the hash simply turns the hash into the password itself. Anyone who has your hash, has your password.

The biggest reason for hashing is that, even if an attacker gets access to the hashed passwords, they still can't use them to log in. Client-side hashing completely and utterly undoes that benefit.

Passwords in transit should be protected by SSL. Not a hash.

You should check out SCRAM. It solves this problem without sending the hash or the plain text.

Fascinating. Yes that does solve the problem, and most importantly doesn't send the hash (unlike what the commenter I was responding to was suggesting, which continues to horrify me).

Most interestingly, it solves the problem where the server may not be trusted. I see how this would have protected against the Facebook problem.

I'm curious, have any major websites you're aware of adopted SCRAM as a best security practice? It feels kind of like overkill, since generally the server is considered to be trusted... but at the same time, it would definitely prevent accidental logging.

I have not heard of any website using SCRAM, but recent versions of PostgreSQL use it for password authentication.

The solution is trivial: a watchdog. Have a watchdog log into a dummy account with a "well-known" password and then inspect logs for that well-known password.

That doesn't necessarily help when logs are sampled. Eg, for high volume services or endpoints, only 1 in X,000 requests might be logged, and you have no theoretical guarantee that you would ever catch any one individual request in your logs.

Good point, though I consider that as just an implementation detail. Bypass sampling for designated system accounts and you will be able to see if systems leak cleartext secrets into logs or not.

But then it's unbelievable that they don't have watchdog for this in place given how big and user-facing that company is. Also out of those 2000 employees having access at least some would have noticed that, wouldn't they? And it still took far to long to correct that.

Why does traffic going through the subsystem contain plaintext passwords?

It was my understanding yhat my password, on a properly configured login page, never left my browser, much less crossed multiple machines that had the ability to read it.

No, your browser sends the password to the server in the HTTP request, only protected during transport by it being HTTPS. On Facebook's side, the HTTPS gets decrypted and the request is passed on (again potentially encrypted in transport, but the machines handling the request obviously need to see what is in the request, which includes your PW)

Ideally Facebook's website or any website should encrypt the password client side and send that via HTTP to the server where the server decrypts in internal and authenticate login session?

Unfortunately, we are in 2019 and client-side hashing is rare because people use SSL instead.

One can argue that company like Facebook that is well stocked with tech resources should have figured this out already but here we are.

Let's say a company does implement client side hashing. Now the server just needs the hash to log you in. Isn't the hash now basically the password? Now if the hash leaks or gets logged, a malicious user can still login to the company service using that hash. Only difference is, since it's not a plaintext password, with proper salting the user is somewhat protected from having their other services with similar password hacked.

Consider this approach: Let H be a hashing function and p be the users password. In the database you store a nounce n, and H(H(p⊕n)). When authenticating the server sends n, and client responds with x=H(p⊕n). Now server can compute H(x) and compare with the stored value to authenticate the client. Finally, after it has been authenticated, the client generates a new nounce n', and sends H(H(p⊕n')) and n' to the server, which is stored in the database for the next log in.

Replace the outer H with a proper key derivation function for extra credits.

This avoids sending any secret value over to the server, so no server side logging will cause a problem.

After reading the comments I have an idea. I'm sure someone thought of it already though. If the password is p and the hash function is H(), server stores the hashed password H(p).

When the login screen loads, server sends the server time so with reasonably fast internet, the client can estimate the server time. Let's call the estimated current server time t. On login, client sends H(H(p)+t) with t. Now the server can compute H(H(p)+t) with the t from the client and verify if the hash match and also check if t is within few seconds of the current server time.

This way if any data that goes over the network leaks for gets logged, it'll only be valid for few seconds. Also salting before hashing should go somewhere in there but it'll make it a bit more complicated.

SCRAM has solved this and with it the server can verify that the client has the plain text password while only storing the hashed password a d without exchanging a plain text password.

FB could also rotate the hash any time, with a short overlap period to allow for already-loaded pages, to eliminate this issue.

No. It would be nice if this was the case but browser sadly just send the password to the servers instead of using something like SCRAM which only requires you send the password when setting it and not on every authentication.