Yes but how many people looked at these logs, saw the passwords, and said nothing? If it was logged presumably the logs were viewed from time to time. The mistake is easy to make, but it's also easy to correct if there is a culture to report such mistakes.

The article indicates it was searchable and used for something by employees, so most of these comments to the effect of ‘they must have been doing this accidentally because Facebook is just so big therefore they are excused’ are invalid. Some group of people did this on purpose and knew it was happening.

You're probably being downvoted for speculation, but that sounds completely reasonable to me. At least one person had to have noticed a password in a log file they were viewing at some time. Most people viewing log files know that passwords should not be there. It would trigger alarm bells, and depending on the person, also excitement -- you have somebody's facebook password.

It's conceivable that someone then told their friend at work, and a few of these 2,000 developers knew of this secret internal stash of passwords they could access whenever they wanted to "prank" someone on facebook...

Why would humans be reading these log files?

For plenty of reasons, I'm sure. From the article:

> My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords.

That doesn't sound like a human reading the log. It sounds like automation.

A script counting the number of successful logins, for instance, could read the same data element that unintentionally contained passwords.

We don’t have enough information to make that judgment.

When you’re developing a feature like this, don’t you look at the data that you’re logging in to make sure things are working properly? I would imagine that Facebook has many layers of abstraction, but in somewhere, incompetence or inattention must be involved, sometimes known as negligence, if not outright knowledge of this. The definition of that is a complex thing, but if someone could have reasoned that this was logging plaintext passwords, and either saw it and didn’t bother to change things, or didn’t think about it carefully enough to realize that it was doing this, it would be considered negligent. I know that I would feel that my trust in this organization has been betrayed, as a user.

The logs being in a searchable index makes it more likely that the password storage was inadvertent, not less. It implies that the primary usage model for the logs was targeted queries, not people starting at the top of the logs and reading down in such a way that nothing could have been missed.

My interpretation was that they were using the plaintext password as one of the searchable fields, presumably for development related to authentication.

That's true. Kerbs article says:

> My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords.

Those numbers are pretty meaningless without some concept of what queries they were making. When I was working with large logs I'd regularly pull huge amounts of logs to only look at maybe a handful of lines. Even then often only scanning for the things I'm specifically working on. Not saying this is what happened just saying those 2,000 employees almost definitely were not doing careful analysis on the complete result of their 9 million queries(And shouldn't have!).

'Only scanning for the things one is working on' to the point of missing huge security holes or safety risks is exactly that's wrong with the industrial division-of-labor approach. Workers start to assume that even if they do notice a problem, someone else is either going to fix it or has examined and OKed the current procedure. Workers who speak up and assert that something they noticed actually looks more important than whatever task they were originally assigned to are typically punished rather than rewarded, and fired if they make too much of a fuss because otherwise managerial authority might be diminished. Most firms run internally like a dictatorship, and given Facebook's absurd ownership structure (where there are voting and non-voting shares Zuckerberg retains a majority of the voting shares) it's a recipe for dysfunction.

This is true but not very interesting. Like literally every large tech company in the entire industry, Facebook has a specialized team of people that work on security --- it has one of the better versions of that team, for what it's worth. Just like ordinary Apple engineers, ordinary Google engineers, ordinary LinkedIn engineers, and ordinary Airbnb engineers, ordinary Facebook engineers don't have an end-to-end picture of how all of security at Facebook works.

There's no evidence that there was whistleblowing suppression about this issue anywhere at Facebook, is there?

Out of 2000 users, there was not one that had a look at username/password? Impressive. Also developers during dev. Seriously, this is not justifiable, it's pure laziness if we want to be positive.

What if one among the 2000 was curious ?

Zuck doesn't believe in hiring employees over 30 because they're not 'smart' about tech. Great example why that's a mistake.

If he did, someone would have seen that with 10-15 years of corporate experience and would have found that simply unacceptable. They also would have had the gravitas to know how to escalate things to the top and gotten it fixed.

I understand that, but I believe they do not have the correct mentality here. I'm not certain how many of the people saying that have worked on systems as sensitive as FB. They think it's easier to screw this up because, for them, it is, but they don't operate at this level.

This sort of thing should not be any easier to screw up when you operate at the scale of FB. If it happens then you have the wrong procedures in place. In my personal example our logs and logging practices/code were audited in the same way our DB layer was. There was no difference; if a system touches sensitive data, it is a massive vulnerability.

This. If you have two or three engineers at a startup it's a very easy mistake to make. With thousands of engineers it's an impossible mistake to make, so it could have only been done purposely.

"could only have been done purposely". In this case there's also a chance it was negligence. Who gives a shit. Move fast. A negligent culture can probably do more damage than the odd individual doing something deliberately.

And even with a startup of two or three engineers, it's pretty much unforgivable.

It depends on how many users you have and what sort of data you're protecting. If you have a billion users then sending password reset tokens to your analytics provider is a very serious vulnerability. If you have 100 users and you're running a generic message board then it's not really a vulnerability because the difficulty of that method of attacking the assets under protection almost certainly exceeds their value.

> that method of attacking the assets under protection almost certainly exceeds their value

This was most likely the reasoning Sony had used (prior to 2012) when deciding how to safeguard the account info of users who had registered for marketing promotions.

It turned out that user credentials are unlike, say, office furniture, in that the cost of a data breach can vastly exceed the "value" of the protected "asset".

That's the best practice in the security world though. There's no such thing as an unhackable system, only systems where A) the cost of attacking them is more than the value of the assets under protection B) where the time it takes to attack the system is long enough for the attack to be detected.

C.f. Bruce Schneier's book: https://www.schneier.com/books/beyond_fear/

That book was from 2003.

This is Schneier in 2016: https://www.schneier.com/essays/archives/2016/03/data_is_a_t...

Data Is a Toxic Asset, So Why Not Throw It Out?

"All this makes data a toxic asset, and it continues to be toxic as long as it sits in a company's computers and networks. The data is vulnerable, and the company is vulnerable. It's vulnerable to hackers and governments. It's vulnerable to employee error. And when there's a toxic data spill, millions of people can be affected. The 2015 Anthem Health data breach affected 80 million people. The 2013 Target Corp. breach affected 110 million.

If data is toxic, why do organizations save it?

..."

I really could not disagree with this more.