This has happened to basically every large web app company. Turn on debug logging in the app which logs HTTP request headers and likely doesn't strip out sensitive information. Easy mistake, and hopefully it wasn't like this since the beginning, and was maybe only found to be an issue for a subset of Robinhood's logs or something.

This happened to Apple too, while logging API errors (CVE-2014-1317 - in this case it was hex-encoded).

Given the frequency that this occurs, I wonder if it'd be a good practice to use a sentinel value for a password during testing and grep logs for it.

> good practice to use a sentinel value for a password during testing and grep logs

Yes. I thought that was pretty common practice. We catch PII (not just password) log-leaks this way that verifying this way often enough that our developers have even learned[1] to check the searches before we yell at them when they try to take something leaky to prod.

If you don't automate detection of things like this you will make the same mistake again.

[1] Not being down on developers, I used to be one and still write a lot of code. Just noting a somewhat less than universal interest in compliance issues.

> Just noting a somewhat less than universal interest in compliance issues.

Is there a compliance document that requires this approach?

Or is it that the compliance states reasonable safety whilst handling PII and security credentials, and that it is down to the interpretation by each individual company as to what that means.

GDPR covers fines for logging sensitive things like passwords. There is additional separate certification needed if you're handling any payment details (i.e. Card numbers) yourself

Any recommendations of tools/libraries that help with this sort of testing/analysis?

So, if I understood correctly is that having a test info like username password etc and then grepping the logs for those known test info values and if it returns anything, then throw an error?

Yes. Ideally this would be an automated process, since it'd be tedious to grep every time you're messing around with logging, and you may sometimes forget to do it. If you're forwarding logs to something like ELK or Splunk, you can set up an alert / scheduled search to continuously check for those canary credentials every few minutes.

Maybe we should avoid sending passwords altogether. With some client-aide code, you could build a challenge-response system that eliminates this risk entirely.

It is possible. But much easier to get wrong than to get right.

Specifically if you hash the password client side then the hash fundamentally becomes the password, and having that sit plaintext in logs is identical to the pre-hash password since both can re-played to authenticate.

I believe this is still a respected/secure version of what you describe:

https://en.wikipedia.org/wiki/Secure_Remote_Password_protoco...

No, having a client side hashed password sit in your logs is way better because it only introduces risk for your own service, instead of risk for every user who reuses their password on multiple services. It changes a compromise log from a widespread identify theft risk to a contained privilege escalation risk.

(Provided either you, or all the other services using this scheme, are sensible enough to salt with a string unique to their website such as a domain name)

just "hashing the password" is not a "challenge-response" protocol.

Thank you

Has there been any research into this or any companies that use it yet?

There's this thing: https://trykno.com/

As they're incredibly opaque about what they're doing, it's hard to tell whether they're part of the "get it wrong" crowd, but being incredibly opaque about what they're doing doesn't bode well.

I used this when I was doing web apps for small enterprises in early 2000's - I sent a random string in a hidden input in the login form and had some javascript concatenate the password and the random string and send me back the username, the random string and the hash of the concatenated string. I hope I got that right (not doing web apps any more since a long time ...).

You would want to use a token and the string and only accept the token to look up the string that you would only accept once and for a limited time.

Otherwise its vulnerable to replay attacks

Yes I think I was cashing the generated string server side for a while and checking if the string I received back had a matching entry in the cache, otherwise reject it.

Anyway, it's been a long time ago, and if I had to do it again today, i would rather not roll my own.

Fido (and now w3c)'s WebAuthn come to mind

Chip-and-pin credit cards use it. 2FA/Fido/Yubikey use it.

Amazon use it in their Cognito offering.

if you can encrypt from the client side, you're always going to be able to reverse it too, its like storing the md5 of my plaintext password, just because it doesnt LOOK like my password in plaintext, it pretty much still is

And the hex encoded version of it.

base64, too, in all four alignments.

I guess the struggle is a) what format its being logged in and b) knowing exactly where things are being logged/stored.

It would be even better practice to use type systems that don't allow the printing of values deemed to be sensitive.

Perhaps users could be asked to use their password-typed keyboard when entering passwords, and their address-typed keyboard for their addresses too?

Not sure if this is a joke, but iOS does distinguish between these.

"Secure" text inputs get the system keyboard, regardless of the user's custom installed keyboard.

That's no excuse, especially for a financial services company like Robinhood. They have a duty to have process and people in place to prevent this exact type of issue.

plain text passwords in HTTP request headers? Wouldn't that be usually in POST body which almost never gets logged?

I have, on a not insignificant number of occasions, used mod_dumpio in Apache to debug things. Which means I end up with all POST payloads in log files. Which has occasionally meant I ended up with plaintext passwords in log files... I am extremely careful when doing this. But I have seen cow orkers copy my debugging techniques and "forget" to switch off the dumpio logging - which has at least once been discovered when a prod server's log files started blowing out in size dramatically. With real user passwords in it... (Quite how mod_dumpio and it's config ended up on a prod server is another rant, which needs to be done over whisky, not in the office...)

Small enough POST bodies could still be worth logging to detect issues. Of course because of things like this it generally isn't a good idea, but there's still instances where it could be important to log POST bodies to eg. understand certain automated attacks.

Headers often include session tokens, though

While careless and problematic, that is less problematic than logging passwords. The user probably reuses their password, so as soon as some developer can just look at logs and get a bunch of email/passwords, they probably now have access to other systems that they shouldn't.

That said, there are more than just logs to worry about. Is your service-to-service traffic encrypted? Then passwords are in the tcpdumps that you took to analyze a strange networking bug (you then copied the password to your workstation to look at the trace in wireshark). Do your programs segfault and core dump? Then passwords are probably in that core dump. Do you check the bounds of every memory read and write operation? Then passwords are probably in random variables (see: Heartbleed/Cloudbleed).

Sessions, though, are less scary. Sure, if you steal someone's cookie internally, you can impersonate that user on your own services. Nobody wants that, because it probably bypasses all the internal auditing systems; the auditing system can't tell that request apart from a legitimate user request. But, cookies can be revoked and have a shorter lifetime than a passwords, so it's not quite as bad.

Ultimately, it's all about limiting risk. If you have a database full of passwords, then someone compromising that database today gets all passwords ever. If you have a disk full of logs full of passwords, someone gets all the passwords that were used to log in within that log server's retention time period. If someone hacks in and starts tcpdumping your internal network and it's not encrypted, they only get passwords from users that log in while they're running tcpdump. If you encrypt network traffic, then someone only gets passwords that were in memory when your binary crashed. Nothing is perfect, but you can do more to add more security. Perfection is impossible, but not logging passwords is one step closer to perfection.

Session cookies should be locked to a TLS channel ID.

Then they are useless to anyone who steals them, since a TLS channel ID can't be recreated except with access to the TPM of the computer which initially created the cookie/session/TLS connection.

Can you post a bit more about how to do that? Your comment is a first result for my googling

That requires support for token binding which google killed.

This is why there is client side hashing. You still hash+salt on the server side, but the client side hashing prevents the worst kind of exposure.

What's the significant difference between sending a plaintext password and a sha256(plaintext password) to the server, except if the user reuses that password on other sites?

> except if the user reuses that password on other sites

Except you can't really except that.

But isn't it bad practice to send plain text passwords to the backend? Shouldn't you encrypt them in the client already?

"This has happened to basically every large web app company."

That is a seriously sad reflection on the poor quality of the software cranked out today.

Someone should be losing their job.

After having the risks explained to them and having received our professional recommendation that they add application-specific logging that had no means of accidentally logging PII, one of our customers (a major bank) still insisted that we implement full HTTP body logging in a proxy.

Nobody is going to lose their job over this because the practical consequences are that you have to send out an "our bad" email and offer a credit protection voucher that most customers won't actually use. There are no costs.

You’re both correct: someone “should” lose their job, but nobody will.

Not that that someone is necessarily the person who actually turned on this logging, it could be their supervisor who let it go live, or QA or… wherever the buck stops.

Sure, people should lose their job because someone's diagnosing a network problem, happens to run tcpdump on a server behind a TLS-terminating load balancer, and sees username/passwords. /s This literally happens all the time.

Depends, how did they get to the server (secure jumpbox)? Did anyone know it'd be a security related event when they were diagnosing the problem? Did the tech take precautions? In such a case, I don't think robinhood.com would need to notify users. They tcpdump would be handled securely, scrubbed properly, troubleshooting done, and that'd be the end of it.

If that isn't what happened, someone should be losing their job. The minute you think diagnosing network problems is a reason to ignore security to a degree that your users need to be notified you're incompetent.

If they used the info to hack accounts or other malicious activity, then yes, they should lose their job.

If they just left a log or capture file around somewhere, nope. Everyone makes mistakes.

"If they just left a log or capture file around somewhere, nope. Everyone makes mistakes."

I guess that's where you and I differ. There are certain mistakes that are tolerable. But ones that result in mass emailing to your clients that they need to change their passwords are not. I'd never heard of robinhood.com before, but now I have, and my impression is I'd never trust them because they have staff that makes mistakes with my secure information.

Think of how robinhood.com's employees feel about this. Management, and the people who are now watching the fallout rob the company of revenue.

Telling me that the person who made that mistake is gone might make me think differently.

The minute you start taking users name and passwords you're in the big boy world and need to treat it as such. Mistakes can be fatal, not just for your job, but your company.

Are you aware that Github, Twitter, and Facebook also logged passwords?

https://www.zdnet.com/article/github-says-bug-exposed-accoun...

https://arstechnica.com/information-technology/2018/05/twitt...

https://www.theverge.com/2019/3/21/18275837/facebook-plain-t...

You're proving my point. It's not taken seriously enough, even by huge companies with lots of resources.

I appreciate their transparency. It should make you trust them more because they're letting you know about the problem.

Some organizations would simply ignore this and move on. You'd never know it even happened.

I'll admit, that's a good point that made me go, "Hmmm."

Blameless post-mortems are a thing for a reason. Honesty about mistakes should make you trust an individual/company more not less.

Just my 2 c

Since we now put the user-id first, before the form goes to check how to even try to auth said user, we can just fetch a strong random salt from the server in it's response "just use a simple password, this is not a fancy account". Some nice, salted PBKDF shouldn't be a problem then.

Also, there are zero-knowledge authentication systems, but they are somewhat iterative due to their probabilistic nature. They are secure in the face of an imposter authentication service, as the password (or a hash, obviously) has to be stored in the real authentication service. But it is never transmitted after this initial store, so no amount of logging or posing as an evil imposter risks disclosure.

I'm not an authentication system programmer, so this may be a silly question, but why do clients still send passwords to a server? Doesn't it make more sense to hash the username and password together with some sort of nonce/salt that's sent to the server for validation?

If you hash the password on the client, then the hash essentially becomes the new password and you haven't solved anything. What you're looking for is a password authenticated key agreement in which one party authenticates to another without an eavesdropper learning any secret [1]. For whatever reason, no major websites use PAKEs today.

[1] https://blog.cryptographyengineering.com/2018/10/19/lets-tal...

I agree that it becomes the new password, but it prevents your own site becoming a contributor to credential stuffing attacks - if you don’t have the clear password transmitted to your systems, you can’t leak it accidentally in logs or through poor DB practices.

I wonder why PAKEs haven’t caught on?

Anything faster than bcrypt is practically clear text due to brute forcing, and running bcrypt inside the user interface is needlessly expensive. The common practice is to ensure that passwords are checked and discarded before they can be accidentally disclosed, or to avoid re-usable passwords altogether. Production logs should also only be accessible when there is a direct need. Sometimes this still fails, but it is relatively rare in well run sites.

The AWS Cognito User Pool Authentication Flow utilizes an augmented PAKE (SRP). I imagine there are a number of major sites that use Cognito along with the SRP auth flows baked into their std libs. I know I've used it a number of times.

I implemented SRP a decade ago — it has issues, and thus a lot of revisions. It also leaks your salt and you can’t use a pepper. There is Opaque (see the play on PAKE! but it’s new and difficult to search for).

But you have. It means even if you refuse your passwords elsewhere, only the website that was compromised is compromised.

You can (and should) store the salt server-side. Moving the hashing to the client does not solve anything.

Really, that should be a salted hadh to really defend against password reuse.

Blizzard Entertainment uses a PAKE implementation (SRP).

This article (somewhat ironically) talks about it:

https://arstechnica.com/information-technology/2012/08/hacke...

I don't know PAKE to know if this is equivalent to it, but you can use a timestamp + hash to avoid sending anything equivalent to a password. The caveat is your system needs to have the correct time set. But otherwise it works fine.

> why do clients still send passwords to a server? Doesn't it make more sense to hash the username and password together with some sort of nonce/salt that's sent to the server for validation?

You must ALWAYS hash the password (as a salted iterated cryptographic hash) on the server. You can in ADDITION also hash the password on the client end, but by itself that's not enough.

The issue, as usual, comes down to "what is the attack you're trying to thwart"? If you hash on the client and not the server, then when (not if) the attacker manages to download the hashed password set, the attacker can create a modified client & send those hashes directly. If you don't hash on the server, it's exactly the same as storing passwords as clear text, because you're storing the data that an attacker can directly use to log in.

You can also hash the password on the client IN ADDITION to the server. In that case, you're hiding from the server the actual password you type in. That's an improvement if you're sharing a single password across many services; in this case the attack you're trying to thwart is to prevent an attacker from actively capturing the password & trying to reuse the password on other systems. However, a much better idea is to not share a single password anyway, so it's not such a great thing.

Users reuse passwords. The main reason of not having your passwords in clear text is to protect them from that. At least when (not if as you noticed) your site is hacked (the same day this attacker download your password database probably), it has no other impact for your users that whatever can happen on your site (the attacker might be able to use the hashed passwords to log on your site but at this point you probably have bigger problems, given that someone is able to download your password database, and you are probably going to reset all those passwords anyway).

This comment is full of many inaccuracies. Leaked hashed passwords still pose a risk to users on their other accounts that may use the same password. Hashing passwords prevents internal abuse and makes your site a much less juicy target (in addition to making leaks less disastrous). It also makes timing attacks far less feasible by enabling reasonable constant time comparison of the hashed password.

I agree with what you are saying but I don't see how this conflicts to what I said.

That introduces additional complexity, and the password is already protected by https.

As others have pointed out, if you go for complexity, there are better schemes. However, by avoiding the complexity you keep simpler client side code.

Especially because this is standard, whilst doing extra stuff would involve doing your own crypto code.

The only thing that you nées is a hash fonction. I think thé added complexity ils negligible on any modern System.

Small tip: looking at the spelling errrors, I bet you're on mobile, and using the standard Google or iOS keyboard set to French. I constantly have to switch between English and my native language as well. But I found out that with Swiftkey (by Microsoft), you don't need to manually switch between two languages; you just configure two language at once and the software does autocorrect for both.

Complexity of the code base. The computation of hashes client side requires quite a bit of custom code. Sending a plain text password is build into the browser.

If you take your idea and step through the login//authentication process, you'll find that what you're suggesting isn't really all that different to what is done normally.

Overview, Scheme_0: - You send your unique identifier (email, username, etc) and password to the server in plaintext. - The server looks up the password hashing scheme and salt associated with your identifier - The server checks that salt+password+hashing scheme produce the stored hash - Server kicks back proper response

In your scenario, Scheme_1: - You prehash the password locally and send your identifier + prehashed_password to the server - The server looks up the password hashing scheme and salt associated with your identifier - The server checks that salt+prehashed_password+hashing scheme produce the stored hash - Server kicks back proper response

The only difference is the extra hashing step. From a security perspective, there is no gain; storing the prehashed_password from Scheme_1 in plaintext (e.g. logs) is no different than storing the password from Scheme_0 in plaintext.

The difference is that the password of scheme 0 is probably reused on many other websites. If for example you bcrypt the password client side, you at least don't have to deal with credentials that might work with all the other websites that your user is using.

How do you bcrypt at an appropriate difficulty client-side? Consider that it probably needs to work on a mid-range Android phone from years ago.

Password reuse on other sites is not your problem. It is your user's problem.

A fuck-you,-got-mine sort of approach to security doesn't work. It's everyone's problem.

PAKEs are probably a good idea across the board.

Take a look at SRP: https://en.wikipedia.org/wiki/Secure_Remote_Password_protoco...

No it doesn't. In your example, the hash of the password becomes the new password (I.e. the new secret). The nonce/salt adds no security because it must be open to anyone attempting to authenticate.

Surely it adds security in that an attacker cannot take that new password and use it on another site, even if the user re-used the underlying password?

Yes. It adds this feature. But only if you assume that the database was compromised or nefarious logging was enabled and no further access to the server was possible. Otherwise, the attacker can modify the JavaScript that is used by the client to perform the hashing (and remove/adjust the hashing function).

To be clear, it would have prevented these logged passwords from impacting other websites if the true cause of the Robin Hood password reset was a logging issue.

As a user, you can prevent this from happening either way by choosing strong, unique passwords for every service.

This is still a meaningful improvement because accidentally making everything world-readable is in many cases easier than accidentally making everything world-writable.

It’s not assuming that has happened, it’s acknowledging it could happen (or any other bug that reveals database field, e.g. sql injection). It seems like it should just be a default practice - generally I try to design systems to fail safe, regardless of what causes that failure.

some additional discussion: https://security.stackexchange.com/questions/53594/why-is-cl...

I've also wondered why this isn't more common.

I wonder if there couldn't be some form of automated tooling which creates dummy accounts in your system and looks for leaks of this type?

That's a really interesting idea. Generate unique tokens e.g. for detecting leaked passwords and alert upon any text match within database, logs, etc.

Azure has this built into some of their DLP...at least when looking at emails

It wouldn't just be for passwords. It would be for any kind of confidential data.

Set an one time url as a password. Someone accesses it bam.

The detector would have to run everywhere, not just on your web application. It'd have to check linux/nginx/apache/etc logs, metric systems transit pipes and storage, etc, etc.

I guess you could install it as a Linux system which searches through all plaintext traversing the network AND filesystems of the entire infrastructure. Which might be a good business idea if you could pull it off.

I agree that it's likely a log file. I remember hearing about Twitter doing this (I also just found out that github may have done this as well).

https://www.bleepingcomputer.com/news/security/twitter-admit...

I worked at a place where on incorrect login, we recorded your attempt (with whatever was in the user/password field) into the logs. The logs went to data warehouse too. That was a fun day in the office :P

Exactly. They've got Kibana or similar setup with uri traces, tonnes of companies have been caught up with this same issue.

Not palantir, their logging system defaults to not writing unknown values.

+1 on that.

I remember on a bank's IVR I was auditing, required "_random 2 out of your 4 characters of a PIN_" (not your card's but dedicated to a call center support - that gave you full access to your accounts though) was being recorded/logged together with any phone key press (e.g. your customer ID). It was possible (if obtaining the logs of a few of a customer's calls) to reconstruct the PIN (e.g. call1#1 you press 1, 3, call#2 you press 1,2, call#3 you press 2,4). Those logs were scrubbed/dropped once a year, so for a frequent IVR user it was easy to reconstruct your IVR PIN. The folder containing the logs was a Windows shared folder with Everyone/Everyone (share/modify).

You should have seen the COO's face as I was describing how easily this could have been abused (likelihood), and how it could open a floodgate of serious problems for the bank (impact).

This is what I thought had happened.

They've supported MFA for awhile, so simple enough to reset a password and make sure MFA is turned on.

I prefer when companies are proactive.

Perhaps it would be helpful to somehow modify the title to reflect this. It doesn't seem at all that Robinhood is being negligent here.

I would agree. Although technically if they were accidentally in a log (the likely scenario) that does mean they were being stored.

The distinction still matters and I give Robinhood credit for admitting it and immediately notifying their users.

Security is a long term investment and it's easy to mess up. Taking responsibility and being transparent when you don't is the only way you're going to get it right.

There are some reverse proxies that are being sold as a way to mirror HTTP data to metric/analytics systems, which would bypass built-in filtering mechanisms which automatically remove passwords from logs (as Rails does). Metric/big data systems are an easy place for sensitive data leaks to happen. So your web developers could be getting security 100% right but some higher level systems integration with an analytics or ops software (requested by marketing or BI guys or whatever) could mess it up.

Logs or audit trace is the first thing I thought of as well. Have seen that happen multiple times. And yeah, logs are stored somewhere so the headline still makes sense (logs are "stored" on disk). Not that it makes it a good thing, but it's a bit of a different story than what is implied.

Yep, probably the same thing that happened with Facebook recently: https://newsroom.fb.com/news/2019/03/keeping-passwords-secur...

good point. it would still be silly for 'industry standard methods' to log passwords, in plain text - even for debugging purposes. But it could be result of some bug or error which dumped a load of information ,or perhaps even crashdumps of processes. they contain memory ,which could contain passwords if for instance, the login handling service crashed. It would be useful if companies were a little more clear about how these things happen, it'd be a good learning point for readers with similar setups in their environments. since the issue was resolved, it could be 'responsible disclosure'.

The version I got said the following:

>>> When you set a password for your Robinhood account, we use an industry-standard process that prevents anyone at our company from reading it. Additionally, our policy is to not store usernames and passwords for any bank accounts that you link to your Robinhood account.

On Monday night, we discovered that some user credentials were unintentionally stored in a readable format within our internal systems. We wanted to let you know that your Robinhood password and linked bank account credentials may have been included. <<<

What level of incompetence is required to have accidentally logged bank account credentials?

This is not simple logging of login, oops, this is a very serious breach of basic security.

My first guess, too. I flagged the story for the misleading headline, but alternatively an admin could change it to "Robinhood security incident" or something to that effect.

> I flagged the story for the misleading headline, but alternatively an admin could change it to "Robinhood security incident" or something to that effect.

Why is the headline misleading? I mean, they say in black and white that they stored passwords in a readable format. The headline is completely factual.

If my password gets compromised, it's irrelevant where Robinhood stored it, just that they did so in plain text. That it was stored in text files, in a database, or handwritten on pink post-it notes doesn't make much difference.

And this is why I still use an old framework like Ruby on Rails in 2019. Highly opinionated, and old, but rock solid and not easy to screw up.