Then some devs miles and years away didn’t use that feature properly and accidentally failed to not log passwords in an incoming request. They may not even have been looking at those request logs because that’s not the request they were testing.
Then that feature went into production and this oversight was magnified millions of times.
At large scale you don’t just tail the production log firehose and look for stuff. You have to search for specifics to find anything st all. So if nobody was debugging this thing in production it’s quite plausible nobody saw the passwords in the log.
One way to catch this sort of thing is sentinel data — in this case, have a unique value for a test account’s password and test every service with it, then search everywhere you can think of for that value.
The notion that "it could easily happen" that is being brought up throughout this thread should really only suggests that people aren't doing even rudimentary security assessments (or, hopefully, they're not working with security sensitive software).
If you can't solve it technically, you solve it through processes and training. Same goes for any other industry -- if a construction worker said that it's just one bad morning away from dropping a two tonne girder on a playground, we would never accept that. Or a pilot crashing an airliner into the waiting hall when they're supposed to land. Somehow it seems that large parts of the software industry simply hasn't reached the level of maturity we expect from pretty much all other industries.
Facebook is an enormous company. They should be able to have entire departments working on these topics. It's not a one-person hobby project we're talking about.
>Somehow it seems that large parts of the software industry simply hasn't reached the level of maturity we expect from pretty much all other industries.
True, but that's a rather broad brush — in terms of actual risk of damages there is nowhere near an equivalence between "airliner crashing into waiting hall" and "logging some plaintext passwords".
Of course the culture, priorities, and domain are also very different between social network engineering and airliner engineering, which is by the way one reason Facebook could grow from nothing to mind-bogglingly gigantic in a decade, while it takes a decade to get just one new airliner into production.
Of course things fail and people screw up. What I don't agree with are arguments along the lines of this just being a slight oversight, and that those can easily happen. It should require serious failure on multiple levels for anything like this to happen at that scale, if they are implementing things properly, not minor oversight.
I didn't intend to imply it was a "slight oversight" — it's clearly a significant oversight — but there are people saying it's obviously gross negligence because how could this ever happen in a company that wasn't completely incompetent, etc. No, terrible accidents can and do happen even in companies that are trying hard to do a good job. Just like when a 737 crashes, you shouldn't assume Boeing is totally incompetent, but rather that several things must have gone wrong at once.
Precisely.
Storing passwords in a plaintext DB heppes in the same manner-a dev is lazy or ignorant of the security reprocussions. Which is what happened here; barring evidence this was done maliciously we can assume that this was accidental. But that doesn't make it any more excusable, since it should be clear that logs can also contain sensitive data that needs to be protected/anonymized.
Considering how selective FB is for hiring, I would hope we could expect a higher standard.
If someone logs your password in plaintext but has it encrypted in the database that’s grossly negligent.
Logging like this can easily be attributed to an accident. The person who implemented this logging should get hit with some repercussions because he surely tested the logging and must have seen the passwords when glancing by eye. But other than that, this was clearly a minor oversight.