Same same but different.
I don't know anything about FB's infrastructure, but when I was lead I would have viewed leaks in logs as far worse than something in the DB because our DB's were harder to gain access to.
I get what you're saying, but it's irrelevant. Easy to screw up, hard to screw up, doesn't matter; just don't screw up because the result is the same. This stuff is security 101. If you're logging requests then you need to ensure they don't contain sensitive info.
But yes, either way the end result is the same and you shouldn't do it.
It's conceivable that someone then told their friend at work, and a few of these 2,000 developers knew of this secret internal stash of passwords they could access whenever they wanted to "prank" someone on facebook...
> My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords.
A script counting the number of successful logins, for instance, could read the same data element that unintentionally contained passwords.
> My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords.
There's no evidence that there was whistleblowing suppression about this issue anywhere at Facebook, is there?
If he did, someone would have seen that with 10-15 years of corporate experience and would have found that simply unacceptable. They also would have had the gravitas to know how to escalate things to the top and gotten it fixed.
This sort of thing should not be any easier to screw up when you operate at the scale of FB. If it happens then you have the wrong procedures in place. In my personal example our logs and logging practices/code were audited in the same way our DB layer was. There was no difference; if a system touches sensitive data, it is a massive vulnerability.
This was most likely the reasoning Sony had used (prior to 2012) when deciding how to safeguard the account info of users who had registered for marketing promotions.
It turned out that user credentials are unlike, say, office furniture, in that the cost of a data breach can vastly exceed the "value" of the protected "asset".
C.f. Bruce Schneier's book: https://www.schneier.com/books/beyond_fear/
This is Schneier in 2016: https://www.schneier.com/essays/archives/2016/03/data_is_a_t...
Data Is a Toxic Asset, So Why Not Throw It Out?
"All this makes data a toxic asset, and it continues to be toxic as long as it sits in a company's computers and networks. The data is vulnerable, and the company is vulnerable. It's vulnerable to hackers and governments. It's vulnerable to employee error. And when there's a toxic data spill, millions of people can be affected. The 2015 Anthem Health data breach affected 80 million people. The 2013 Target Corp. breach affected 110 million.
If data is toxic, why do organizations save it?
..."
Or if you think a client bug is a stretch: If you're running a large enough website which logs usernames on failed logins, I bet you have at least one password or a concatenated usernamepassword in your logs. Just because someone wasn't paying attention to the text box focus.
Free startup idea: build a SaaS for the bot phase of this, and make the log-scanner simple and open source so enterprise security teams can deploy it freely. Offer a certification process and consulting. Hire lobbyists (and folks with inroads into insurance companies) to paint plaintext passwords as the devil and make yourself a de facto legal requirement. And make sure your log-scanner doesn't do any logging of its own :)
You're talking in hypotheticals. Nowhere did I say "nothing ever goes wrong if you know what you're doing". What about the issue at hand? What's your opinion on passwords being logged on every request for years?
If what you meant by that is "best effort" then sure. The issue at hand is only obvious now that we're reading about it. It could easily be missed depending on how their infrastructure works.
No, it's not. It's completely obvious to anyone who has ever worked on a system which deals with sensitive information. It can only "easily be missed" if you're not auditing your code and logs. This stuff is basic to anyone who works in industries where the data is a liability.
I find it hard to believe this is accidental.
This explains a lot of the comments here. Facebook’s scale is not like anything most engineers have worked on. Facebook probably has logs in the 100s of terabytes. Ensuring that sensitive data isn’t logged takes more than some occasional greps.
Collecting data on such massive scales is literally FB's whole business.
But with that also comes a responsibility that shouldn't simply be waved away with "But they are so big, it's so difficult!"
Because when it's about monetizing their massive amounts of, often illegally collected, data then FB seem to have no issues having everything in order and getting stuff to work, regardless of how "difficult" it might be.
Probably has to do with the fact that there's no money in protecting users data properly and FB seems to be pretty much immune from negative PR having any bad consequences.
Of course at FB scale you'd automate this by creating a set of canary accounts with unique passwords that you perform a search for in the ETL pipeline, or some other handy place. This will at least catch inadvertent plaintext password logging.
>Ensuring that sensitive data isn’t logged takes more than some occasional greps.
Right; it requires investment into process, requirements, testing, and oversight. Most importantly, it requires a company wide, top down mentality that your customer's privacy and protection is more important than your margins.
If you can't (won't) dedicate the resources required to ensure your customers data is protected then you have no business operating at such a scale.
Treat logs as liabilities: why can't you solve the issue I experienced yesterday?
You can't win. Either you log more than you think you need right now or you can't do engineering investigations on past data. You're going to end up somewhere in the middle realistically.
Unsafe logging of PHI would be explicitly problematic, but passwords aren't PHI.
The analogy was intended to focus on practices which should be employed by companies which handle sensitive data, not the specifics of what the FDA is looking for.
You work in the industry so you know that the FDA is not highly prescriptive. They give you a general outline of, essentially, process. HIPAA goes into more detail, but still, an organization has quite a bit of leeway as to how the meet regulatory requirements.
As far as passwords go, you're correct. Passwords in logs aren't a direct violation as far as I'm aware. If e.g. an employee gained access to data they shouldn't have access to per internal policy and used that data in an illegal manner, _then_ you have a problem.
Still though, I never meant to imply that what happened at FB would violate FDA regs. What I said was:
>I've worked in healthcare / biotech for more than a decade and I can promise you that the FDA would see no difference between the two types of gaffes.
The "two types" in question here were defined by the GP ("easy to make" mistakes like logging and something something about database design.) I wasn't referring to passwords specifically, and I don't believe that this is as bad as e.g. PHI just sitting around for anyone to see.
The analogy was meant to convey this; in a health care environment we are required to secure (encrypt) PHI and PII at rest. If we were found in violation of that, it wouldn't matter if it got there via logs or poor database design.
Again, I realize this is not an FDA/HIPAA situation. My experience with sensitive data is in that sort of environment, and I believe the same sort of mentality should be taken by FB in regards to the security and privacy of their users.
Wow, didn't expect that to get so long. My thumbs aren't made for this.
But you're not wrong.
Pile on the regulation (GDPR for starts, more PII protection to follow up, extremely painful fines for failures).
This doesn't come from software developers.
This comes from the SV culture of "move fast and break things."
I went through classical engineering school, and I'd summarize with three categories:
* Newtonian Physics & The Mathematics Behind It (i.e. Differential Equations)
* Ethics
* Technical Communications
Offering engineering services to the public requires licensure. Working at places like Boeing, General Motors, Texas Instruments, and Caterpillar does not (for the vast majority of positions); It just requires an accredited engineering degree.
Note that this is independent of the software industry needing more engineering rigor, which it desperately does.
"Software Engineer" is just a job title.
> Engineering requires licensing and exams through a governing body.
"Software Engineering" doesn't, it's just a job title.
> Can’t have the prestige of the title without the responsibility.
There is no real-world prestige associated with having "Software Engineer" as a job title (as opposed to "Software Developer").
> Software developers want to have their cake and eat it too (power and respect with no oversight and governance).
"Power and respect" is not actually something that is commonly associated with a guy sitting at a desk for a salary.
> Pile on the regulation (GDPR for starts, more PII protection to follow up, extremely painful fine for failures).
You just learned that Facebook stored passwords in plaintext for years and nothing bad happened. In fact, it's hard to conceive that anything particularly disastrous could have happened. Yet, you call for more regulation that will drive up costs for everything. Whole industries could become unprofitable and those high-paid "Software Engineers" would become redundant.
I have the opposite opinion: The fact that computer security is generally poor teaches people how to properly use computers. They need to assume that there are no secrets, because that is the plain reality. No amount of regulation will make computers secure. No certification or best-practice encryption scheme will prevent users from writing their password on post-it notes, or from opening "sexy.jpg.exe", or from answering that call from the "Microsoft Service Center".
The point is that the public should have confidence in engineers. Software developers who tout themselves as "software engineers" actively piggy-back upon, and undermine this confidence. It's malicious, and you should feel bad for defending the practice.
If you still don't see why, "Lawyer" and "Doctor" are "just job titles" but obviously that's a problem. I trust you can see that.
Yeah well, good for them I guess. I don't think anybody else cares.
> The point is that the public should have confidence in engineers.
Why? "Engineer" is a very broad term used in a variety of occupations for very different things. If you're looking at someone's credentials and you're so uninformed that you can't tell the difference between a licensed engineer in a particular profession and a guy that has "Engineer" written on their business card, you're not qualified to make a decision either way.
> If you still don't see why, "Lawyer" and "Doctor" are "just job titles" but obviously that's a problem. I trust you can see that.
I can see the argument for why a medical doctor or a lawyer should be afforded some amount of protection, because they are directly dealing with laymen. I don't buy the same argument for the word "Engineer", much less "Software Engineer". It just doesn't really mean anything. We already have degrees and certifications for when specifics matter.
We focus a lot on testing and I just finished a mandatory ethics workshop. Its a different approach to software than I read about elsewhere. Almost no time spent on algorithms either - bad for interviews, but great for being able to write software that works though.
And like the job title "sandwich engineer", it should probably be derided.
Apologies to anybody working in the honorable trade of sandwich crafting. I love sandwiches and appreciate your work. They bring me much more enjoyment than software.
But seriously, the definition of engineer in regular lexicon is someone who designs, builds, or maintains complex systems. I struggle to think how software does not tick all of those boxes. Sandwiches are a different story.
Perhaps you are confusing engineer with professional engineer? I can see how it might be easy to mix them up if you are not paying attention, but professional engineer carries a different meaning. Most software engineers are not professional engineers but are unquestionably engineers.
Of course, some people get software engineering degrees, or computer science degrees (that term is another can of worms entirely), but that is not the rule.
Only to the extent that all words are meaningless. However, we keep a reference known as a dictionary that helps maintain some consistency around meanings of words.
The Oxford Dictionary defines engineer as:
1. A person who designs, builds, or maintains engines, machines, or structures.
2. A person who controls an engine, especially on an aircraft or ship.
3. A skilful contriver or originator of something.
Is software an engine, machine, or structure? That is debatable, although I would suggest that it does meet the definition of machine. Under that suggestion, software engineer is a perfectly appropriate term. Designing, building, and maintaining is exactly what software engineers do.
If you disagree that software falls under the definition of engine, there is still that third definition. Is software created by a skilful contriver or originator of something? I think that is a definite yes.
> There has to be some standard, some governing body, some degree of rigour applied here.
Professional engineers are expected to display those things. That is unrelated to engineers. Different terms with different meanings. Software engineers may not be professional engineers, although they can be.
So we should call car mechanics "mechanical engineers"? And maintaining structures? With all due respect to the janitors out there, their job is vital to society but it's not engineering.
By definition, I don't see why not. I suspect the spirit of maintain may be a little more nuanced, but words are fluid and if someone wants to interpret it that way, then it is so.
> With all due respect to the janitors out there, their job is vital to society but it's not engineering.
If by janitor you are thinking of someone who sweeps the floors, I think that is well beyond the spirit of maintain. That, to me, is cleaning.
Maintenance of a structure is more like addressing a beam that is cracking. I doubt that is a role that an average janitor deals with, and in many cases will legally require a professional engineer to spec the resolution.
But words are fluid, so if you interpret a janitor as someone who maintains a structure, then I guess engineer fits. It really makes no difference either way.
Title only engineers work on complex systems.
This distinction has largely been lost and most "Software Engineers" think they are doing PE Engineering when they are not.
Seems a bit silly.
Logging like this can easily be attributed to an accident. The person who implemented this logging should get hit with some repercussions because he surely tested the logging and must have seen the passwords when glancing by eye. But other than that, this was clearly a minor oversight.
Then some devs miles and years away didn’t use that feature properly and accidentally failed to not log passwords in an incoming request. They may not even have been looking at those request logs because that’s not the request they were testing.
Then that feature went into production and this oversight was magnified millions of times.
At large scale you don’t just tail the production log firehose and look for stuff. You have to search for specifics to find anything st all. So if nobody was debugging this thing in production it’s quite plausible nobody saw the passwords in the log.
One way to catch this sort of thing is sentinel data — in this case, have a unique value for a test account’s password and test every service with it, then search everywhere you can think of for that value.
The notion that "it could easily happen" that is being brought up throughout this thread should really only suggests that people aren't doing even rudimentary security assessments (or, hopefully, they're not working with security sensitive software).
If you can't solve it technically, you solve it through processes and training. Same goes for any other industry -- if a construction worker said that it's just one bad morning away from dropping a two tonne girder on a playground, we would never accept that. Or a pilot crashing an airliner into the waiting hall when they're supposed to land. Somehow it seems that large parts of the software industry simply hasn't reached the level of maturity we expect from pretty much all other industries.
Facebook is an enormous company. They should be able to have entire departments working on these topics. It's not a one-person hobby project we're talking about.
>Somehow it seems that large parts of the software industry simply hasn't reached the level of maturity we expect from pretty much all other industries.
True, but that's a rather broad brush — in terms of actual risk of damages there is nowhere near an equivalence between "airliner crashing into waiting hall" and "logging some plaintext passwords".
Of course the culture, priorities, and domain are also very different between social network engineering and airliner engineering, which is by the way one reason Facebook could grow from nothing to mind-bogglingly gigantic in a decade, while it takes a decade to get just one new airliner into production.
Of course things fail and people screw up. What I don't agree with are arguments along the lines of this just being a slight oversight, and that those can easily happen. It should require serious failure on multiple levels for anything like this to happen at that scale, if they are implementing things properly, not minor oversight.
I didn't intend to imply it was a "slight oversight" — it's clearly a significant oversight — but there are people saying it's obviously gross negligence because how could this ever happen in a company that wasn't completely incompetent, etc. No, terrible accidents can and do happen even in companies that are trying hard to do a good job. Just like when a 737 crashes, you shouldn't assume Boeing is totally incompetent, but rather that several things must have gone wrong at once.
Precisely.
Storing passwords in a plaintext DB heppes in the same manner-a dev is lazy or ignorant of the security reprocussions. Which is what happened here; barring evidence this was done maliciously we can assume that this was accidental. But that doesn't make it any more excusable, since it should be clear that logs can also contain sensitive data that needs to be protected/anonymized.
Considering how selective FB is for hiring, I would hope we could expect a higher standard.
If someone logs your password in plaintext but has it encrypted in the database that’s grossly negligent.
It's the same reason when there are large outages, where the comments are split between the enraged customers, and then the ones that are "Man, sucks to be them" as know they could easily have been the one that got the config push wrong.
All in all it just paints a picture of an irresponsible company where their house is not in order.
Those things are in no way exclusive. Storing passwords in plaintext is awful practise. Including the contents of form submissions in logging is a similarly awful practise. I really don't see what the difference is.
>Because there's two types of storing passwords in plain text. There's the "your password is stored in plaintext in the database" way which everyone agrees is 10 different kinds of stupid, and then there's the "we accidentally logged the body of all requests that went through this system, and it turns out login requests came through here" kind.
Basically the conscious decision to store passwords in plain text is worse than unintentionally doing so in logs. One is purposeful, one is not. Yes it's bad, but it's not as bad as if it was done on purpose. And generally, this how most laws are enforced or implemented.
I think people are drawing a distinction where one doesn't really exist.
This whole conversation is about how it likely was not done on purpose, and that it was the result of logging HTTP requests and responses.
If they wrote logger.info(password), yea I'd say that's as outrageous.
I'm also working on a Django project but we're not logging HTTP calls arguments there. I think we could use a filter like [3] but I'd rather have the framework to automatically take care of that.
And if I'd be writing a web app from scratch, no matter if I've been doing web for 25 years, I'm sure I'll do a lot of silly mistakes. That's why I prefer to build on top of frameworks.
[1] https://guides.rubyonrails.org/configuring.html#rails-genera...
With password storage everyone knows the patterns, or we expect them to.
With everything between the request and password storage, we don't.
This type of attack could easily be prevented. When secrets come in, immediately store them (ideally at the web framework level) in a type that overrides print/debug formatting. Then add a "get_raw" to it, and you can now grep for that being used anywhere outside of storage to a DB (and your DB libs should take the Secret type too). Or don't use a `get_raw` and instead use a `hash` method that returns a safely hashed version of it.
Further, your secret type could at the very least add a round of SHA256, maybe even with a pepper?, just to be sure.
This isn't hard, I've done it before. The problem is that it isn't something that people feel embarrassed not to do, vs storing plaintext creds.
Impact is the same - creds are plaintext in a DB. Attackers always expect sensitive data in logs, so it isn't as if you'd get lucky and they'd miss this.
But, in practice, a lot of this stuff is being logged by things like proxies that could be several layers in front of the "web framework level".
Either way, yeah, you're right - it is not a perfect solution. But it means that for any system your engineers build, so long as they build them using your web framework that imposes this password type, you can grep your codebase for bugs.
You could implement client side hashing as a best-effort "in transit" mechanism but that has obvious downsides. Not sure how I'd feel about that approach in practice, but I can't see a big downside.
It's either ametuer hour over there or the organization simply doesn't care enough to invest heavily in the protection of their users.
It was easy to overlook, since as you said, it was intended to log something else (failed API requests), it only happened in the case of an error, and it was hex encoded. I only happened to stumble across it out of curiosity.
Also, though the flexibility of being able to use a different ID for iCloud, home sharing, iTunes, App Store, Messages, etc. is neat, it's pretty annoying to need to set it in each of those places. (And TBH it seems like being able to share purchases/access/etc between IDs is more useful than being able to have separate ones, yet the iCloud family stuff took a while to arrive.)
The issues I've seen aren't as bad now as they used to be, but they haven't left a good impression.
This is not an excuse. If you're logging all request data, you need to strip or encrypt sensitive information in that request data. Handling Persistence of sensitive data is web development 101. Just because it's not in a database doesn't give you a pass to leave it unprotected.
This level of incompetency is unacceptable.
Both are security fails, regardless of the cause of the fail.
And a company the size of, and with the resources of, facebook, most certainly should not get a pass for the "oops, we logged more than we should have" cause. A corp. of their size, and with their resources, should be doing password handling correctly, every single time, no exceptions, no excuses.
Imho, a company (especially as big as facebook) should have the right process and procedures to prevent these kind of problems and ensure developers have proper training to make them aware of the consequences of their actions.
I guess part of this is that passwords aren't considered that important to many people :(
I designed a clinical testing platform a couple of years ago. Our initial requirements stated very clearly that PHI and PII were not to appear in logs. This is basic stuff for anyone who actually works at this scale / level of sensitivity.
What? I absolutely expect every company to not log my password in plain text. In my 15 years as a developer across several companies and industries, I have never seen anybody log passwords, or advocate for logging passwords.
I'm struggling to think why any employee of any company should be able to view a plain text password in any form. Why would there not be an expectation here?
The parent was taking about expectation about infrastructure that validates this. This is both very uncommon and impossible to do 100% correctly. You can scan logs for a prefix (password=), you can do entropy counting, you can try to decode hex values in text. But if you find a base64 encoded hex string representing "foobar" - how do you even know it's a password?
Short of trying all possible decodings of all possible substrings against your full password database, this is an impossible task. (You can do best-effort things though)
nope.
there are n types of storing passwords in plain text (including storing them in memory).
you just named 2.
all n types fall under infosec responsibility to prevent, and none are acceptable. there is no pass because logs were less intentional than storing in a table
So I agree that these actions are indeed same but different.
They ultimately speak to a failure at Facebook, but it doesn't speak to utter incompetence at Facebook.
So many ways this could have gone down, so easy to do.
Sure, then the question for anyone at facebook would be from the Bobs: "What exactly is it, you'd say, you do around here??
FB claims to have "top minds" in essentially every discipline...
Heck, they are IMO a revolving door to the .gov/nsa/infosec community...
So... I call BS on your statement as "whoops! Easy mistake!"
WTF is it that you'd say you do around here, Mr. FB-Security-Guy??
I wonder if it's time to up our game again and go back to that model.
Its only an easy mistake when you're vacuuming up everything people do.
They're both issues with which any moderately competent engineer would be very familiar. Heck I've seen commercial contracts that specified that audits be done to ensure no password and PII leaks into log files. Everywhere I've worked in the past 20 years that logged stuff conducted audits on a regular basis to check that sensitive data wasn't being logged or in inadvertently stored in a database.
The grey area might be for example when some data gets logged as a blob of HEX that it turns out has a password in it if you run the right decoder function on it. I've seen cases like that though show up in audits -- you see a blob of HEX or MIME and go find out what might be in it.
Anyway, it really isn't accurate to say that this is a kind of <slaps forehead> <doh, why didn't we think of that> thing. It gets thought about all the time.
All the transport encryption and DB encryption/hashing/salting won't protect you from this kind of logging mistake, but the above would.
P.S. There are ways to make the above even better by adding a nonce that has to be requested from the server before POST etc.
If you hash your users' passwords using a key-derivation algorithm on the client-side, each user's password simply becomes the original password's digest. From the server's perspective nothing has changed. Moreover the server will need to re-hash the password digest sent over the wire, because if the server is compromised the password digests can be directly replayed to the server to compromise corresponding accounts.
Additionally, since the shared secret between the client and server is the user's password digest, the password needs to be hashed using the same salt every time the user authenticates. So each user's password digest is still a de facto unique password which will be sent over the wire anyway. This scheme basically retrofits the server's job onto the client with added complexity. It's like the (slightly) faster horses version of password authentication, when we could really be experimenting with developing cars (like two/multi-factor authentication, more robust server-side controls and provable correctness).
That's not to say the scheme has no benefits whatsoever. It does mean that user passwords will be more complex, because the actual token stored by the server (the de facto password) is a digest. But there are two drawbacks - with enough users you'll still see many duplicated digests in your password database, even if you randomly generate salts on the client side. More importantly, you're offloading hashing to the client side in JavaScript. JavaScript can be very fast in 2019, but companies like Facebook and Google still maintain very low latency, substantially stripped-down versions of their websites[1] because client-side hashing isn't going to be nearly as fast as server-side hashing for a huge number of people. There's also a sizable population of people who don't even have JavaScript enabled, or who might have incompatible browsers.
tl;dr - Client-side hashing is not a best practice (and not widely deployed) because it comes with a nontrivial complexity increase, lower client compatibility and negligible security benefits. It also would not have prevented this vulnerability.
_______________________
1. For example, mbasic.facebook.com.
Client: asks server for nonce
Server: sends nonce
---- OR ----
Nonce arrives with login page
Client: sends HMAC(nonce + HMAC(username + password + appname) + Unix Epoch rounded to last 5 min block))
Server:
1. gets response
2. using username as key pulls HMAC(username + password + appname) from DB
3. Computes HMAC(last nonce sent to username + DB HMAC + Unix Epoch rounded to last 5 min block)) and compares to user token
4. last nonce is cleared
This kind of gets to the heart of what I was referring to when I said client-side hashes are like faster horses rather than cars. If you're spending this much effort, a superior protocol is better than an unorthodox, modified one. SRP is a PAKE which basically takes your proposal and moves it into a different layer of abstraction (TLS), and OPAQUE makes improvements upon it which allow you to use elliptic curves[1]. There are other reasons not to use PAKEs, but they're a much more coherent and defensible suggestion than just bolting the key derivation system onto the client rather than the server.
______________________
1. https://blog.cryptographyengineering.com/2018/10/19/lets-tal...
Server sends nonce
Client sends HMAC(nonce + password + time)
There are better ways to implement this idea, like SRP/PAKE https://en.m.wikipedia.org/wiki/Secure_Remote_Password_proto...
Unless you actually mean some sort of challenge response scheme which is rather uncommon to see, e.g. http “digest” authentication or SRP.
So it's bad practice to keep or transfer the users cleartext password. It should never leave her browser/client. Period.
This situation, on the other hand, seems to be more of an unintentional capture of passwords by their logging system. That's much harder to notice and more akin to a bug or security vulnerability, which is something pretty much every system suffers from sooner or later.
In such cases I'm more concerned with the company's response to the vulnerability than the fact that it exists at all. Here it seems like Facebook noticed the problem and proactively fixed it before it was exploited, which to me is a good sign.
Logging request or response payloads without an explicit whitelist should raise flags for any developer. There are very few cases where you can assert that not only in the present but also for all future use cases of a system, the entirety of a payload will not contain sensitive user data.
Only a whitelist will suffice to maintain good security. It's common for developers to attach sensitive data for debugging and other use cases under arbitrary paths.
Systems can improve further by adding patterns and other heuristics to drop values from the whitelist that look like sensitive data.
Log in and sign up pages -- especially on mobile web -- are being constantly iterated on by "growth" and "emerging markets" teams whose first priority is getting graphs to go up and to the right, not making sure the pages are secure. Their entire mission is to make things work with yesterday's technology (feature phones, old Android tablets, and so forth).
On the other hand, the dedicated security folks are focusing on ensuring there are multiple factors protecting high-profile targets, so they're working on SMS as a second factor, Login Codes, Security Keys, and primarily focused on Desktop Web, iOS, and Android. Or they're doing fuzzing, or building robots to chase down engineers for buffer overflows and the like.
Ask any HIPPA compliant company how they scrub their logs from errors that include medical data and you'll truly see how bad things are.
Fwiw none of this is necessarily a breach of hipaa laws.
I've never dealt with medical data, but do deal with credit card data so have to deal with PCI. With credit card data we run into customers who send emails, or use online chat, or leave voice mails telling us they have a new card and giving the number. We never ask for them to do this, and indeed everything we tell them says never to send us a credit card by such means, but they do it anyway.
That brings all those systems into scope for PCI, which is a pain in the ass.
We use helpdesk and support chat software licensed from a third party. We had to write scripts that understand its DB schema and data formats and can find and remove credit card information, and keep them up to date as the helpdesk and chat software is updated.
A few years ago, the vendor tried to discontinue the stand-alone version and move its users to their cloud service version, but had to drop that plan when they found out that a lot of their customers were doing the same thing we were, and absolutely could not move to a cloud service unless that cloud service handled PCI issues. Apparently they didn't want to deal with making their cloud service handle that, and there had been no talk since then of dropping the stand alone version.
Do people dealing with HIPPA run into similar issues?
"In this situation what we’ve found is these passwords were inadvertently logged but that there was no actual risk that’s come from this. We want to make sure we’re reserving those steps and only force a password change in cases where there’s definitely been signs of abuse."
Inadvertently logging passwords is a risk. If those logs were accessed then that's a bigger risk. Signs of abuse is an issue. There is no such thing as an "actual risk", there are just probabilities (and possible consequences). Once a consequence happens, it is no longer a risk -- then it's actual.
Early FB (c2005) could get the excuse, new team, just getting started. But, no excuse after the first round of funding
so when you say, "the community usually scoffs at entities which store passwords in plaint text. Why do you give Facebook a pass" is a fallacy of composition.
Facebook is supposed to be hiring the best developers. It has tons of money. This is the most basic security consideration - hashing passwords on the client at least. At LEAST.
I am constantly surprised by how basic practices were ignored by these corporations that got so big, while the small guys implement them. I guess people really were "dumb fucks" to trust Zuckerberg with their passwords.
Edit: You've made this comment twice in these threads. It's wrong.
This community usually scoffs at entities, which store passwords in plain text. Why do you give Facebook a pass?