That doesn't mean we should be letting Equifax off the hook. But nobody should pretend that RCE on an on-prem webserver wouldn't be game-over, for the entire internal network, across the majority of the Fortune 100.
We have as a society chosen to trade the security of our personal information for greater and cheaper access to products and services. Nobody on HN will like that fact (at least, as stated bluntly like that; plenty of them do like the increase to their earnings capability that results from that tradeoff), but it's generally true.
You certainly have more experience with this than the vast majority of the rest of us, but with that kind of data on millions of people, shouldn't it be, and why isn't it, standard practice to put servers storing that data behind an extremely limited API, with a firewall in place to monitor queries and responses and severely rate-limit or stop additional traffic if something suspicious happens, until the monitoring people can take a look?
Those with data on Equifax's servers aren't Equifax's customers, so other than "not having a PR cluster&#@_" economic incentives are not aligned to prevent this.
Also... even with multiple layers, what do you think the backend stack would be? Unless it's built by a separate team deliberately using a different technology platform, you're stuck with the same Struts (or whatever) vulnerability.
The system/network I inherited at new job it absolutely would be, it won't be in a year though.
We aren't a fortune 500 but there are simple, practical things you can do to limit the damage an RCE on a webserver can actually do.
But I disagree, security teams can and should manage large networks with large numbers of stakeholders.
If they can, why don't they?
Insufficient incentives, either caused by inadequate regulation or a belief the regulation won't be enforced.Think of the standards of network security and software validation used by, say, a Certificate Authority to keep their root certificates safe. Or by Google to secure GMail. Equifax just hasn't made reaching that standard their highest priority.
:)
Google would be most people's exception to the rule I'm talking about here, having as they do one of the world's greatest information security teams. And Google has had major incidents!
What's needed is an architectural security review, which is even rarer than people conducting an annual pentest.
You're too optimistic :(
I'm not saying you should like the tradeoff, or that the existence of the tradeoff doesn't have implications for how we should regulate banks and credit ratings agencies. I'm just saying that if you think there's some other CRA that has reliably protected its IT assets from threats like this, you're wrong.
this seems like goalpost moving. first you said 'we as a society', yet there isn't much 'we' in this. it doesn't seem to me that people (the general public) are properly aware at just how vulnerable they are. they haven't therefore 'decided' that this is okay.
> the expense that would go in to making a large enterprise network reliably secure would be extreme, and would drive prices up, retard economic growth
what is this based on?
To more directly answer your question, though, consider a regime of general commercial liability for security incidents caused by failures of engineering (which Equifax is an example of). What would it be like to bring new products to market under such a regime? How much more difficult would it be to enter the field? How much more difficult to hire qualified people? What would that do to the prices of things made possible largely through software engineering work? When the prices of things increase, they're put out of reach of more consumers; markets constrict. What does that do for employment? What does reduced employment in a sector do to wages?
I'm not saying this to justify insecurity. I'm ambivalent about the tradeoffs our society has made on these issues. I just don't think we tend to grapple with them honestly on HN.
My point is: the most savvy companies contract annual sitewide pentests, in addition to all the other stuff they do to secure their networks, and virtually all of those pentests "succeed" (from the perspective of the testers).
Further: not only have I never personally been on an internal network pentest (the kind where you start with an IP "behind the firewall" and nothing else) that failed, but I'm not sure I've even ever heard of one that failed. I'm certain there are some networks that are so resilient that they've stymied a competent internal pentest --- but again, I wouldn't know which one, because I've never heard of that happening.
Your car kills people? Your biscuits are mouldy, or full of lead? We have strict rules, and companies that don't follow then end up in serious trouble.
And they aren't perfect. They will eventually screw something up.
Comparing Stripe to Equifax is unreasonable. Equifax has about 10,000 employees. Stripe has less than 1,000.
And, once again, please don't try to win an argument by putting me in the position of defending Equifax. That's not what I'm doing. My point is that practically every company of Equifax's size has similar problems.
Apple, Facebook and Google have far more data under their control than Equifax and they seem to take security far more seriously than Equifax. The Apple InfoSec team is extremely aggressive — InfoSec is ingrained in Apple culture. I am skeptical that Equifax takes security as seriously.
Being “big” or “other large companies have the same problem” isn’t an excuse. They have no problem fucking people’s credit by reporting inaccurate information — they should be held accountable since that information has such severe consequences.
My hatred of Equifax runs deep because they reported 7 state tax liens on my credit report that weren’t mine and were from a state in which I never lived. Despite providing all kinds of documentation, I ultimately had to sue them. I won the case and a bit of money, but the point is, they were willing to have a team of lawyers go to court against me and tie it up in court for several years — to defend something they knew was wrong. I even had letters from the state tax agency attesting to the error and they still had to be sued to fix the information. If their IT security is as arrogant as the rest of the company, then this breach doesn’t surprise me.
How is that relevant? It’s highly relevant because they have demonstrated a willful disregard for facts. They demonstrate an arrogance based on their near-monopoly power over people’s lives. They don’t care about accuracy — they care about the appearance of accuracy. It follows that they aren’t good stewards of the information with which they have been trusted.
The fact that they are too big and IT security is “hard” — that’s an excuse. General Motors might have bad IT security (I don’t know,) but they also don’t control the financial lives of almost every American.
If they are so big that they can’t handle it, then they ought to be shut down.
To be clear, I don’t have a problem with credit reporting. I have a problem with the fact that Equifax is going to survive this. Some big judgement happens, they’ll file for bankruptcy, they’ll reorganize and be right back at it.
Apple is great. I like Apple's product security team and I like Apple IS&T. But so what? I'm not denying that there are 10-20 firms in the world that have world-class security teams. I'm denying that those teams are in any sense an industry norm, even in the technology industry.
Agree, and it's one of the reasons I generally try to avoid using products that are low-cost or free (the other reason being questionable sustainability, regardless of funding). At some point, products need to break even or they will go out of business. As a customer, you are always paying for the product in some form or another. For something that's free, you might be paying with your personal information now... or later if running on a low budget has caused the product developer to cut some corners when it comes to infosec.
[0]: https://web.archive.org/web/20170908214344/https://qz.com/10...
[1]: https://qz.com/1073221/the-hackers-who-broke-into-equifax-ex...
[2]: https://baird.bluematrix.com/docs/pdf/dbf801ef-f20e-4d6f-91c...
I just checked some Equifax domains against SSL Labs, and while their Canadian site (https://www.econsumer.equifax.ca) scores an A-, it has no forward secrecy. I'm surprised to see a modern web server not supporting FS today. Worse, the main entry point to their Canadian site (http://www.consumer.equifax.ca) as indexed by Google does not redirect to a TLS enabled page, although they do seem to have a TLS endpoint for that domain -- but not sure how people are expected to get to it.
Edited to add: The first link is only accessible through a redirect by clicking on the "Get Started" button on their main Canadian site. Furthermore, even selecting Canada from the drop-down on https://www.equifax.com/personal/ redirects to the insecure non-TLS site.
https://observatory.mozilla.org/analyze.html?host=equifaxsec...
Now a different thing is that should this information be available through a server that is accessible directly through the Internet (and I don't know if it was) or if they should require for example a dedicated VPN connection.
Can you please link to your preferred tutorials (or other sources) which explain how to set up the most secure server architecture, with regards to this story/subject, for those of us who would like to do a better job?
Thanks a lot!
You can do layered security, but how? Think about a customer service rep. They need access to ~everybody's data at a moment's notice because they're a CS rep. That's what they do. That means every CS rep machine has access to all the data.
I'm not sure this is solvable. SSN's are the problem, not the way that we store SSNs. They still need you to read off your SSN when you call in, so you can't just encrypt it.
Think about Equifax in particular. They're a credit bureau. Of course they need all the data available all the time -- that's what a credit bureau does. The question is, how would you partition the data so that a failure in one place wouldn't lead to exfiltrating all the data everywhere?
The trouble is, it wasn't just "they popped Struts and then ran an scp." Even if you partition the data, they popped Struts! They have RCE inside your network. It's absurdly easy to pivot from one machine to another in a given network. You keep a helpful ~/.bash_history file that says exactly what's important and where everything is installed. Stuff like this is why I argued for deleting it, since it was the #1 most helpful tool I had that helped me pivot inside a target network. But then you get people yelling about taking away their bash history.
You can imagine how terrible this problem becomes in practice. Nobody here gives a crap about Equifax; they just want their heads to roll. Unfortunately you'll be next on the chopping block because no one has solved these issues. As soon as the world goes crazy over the lack of security, it's game over for us all because it leads to a highly regulated world that won't be much safer.
See how resistant we all were to the mere mention of increasing security procedures: https://news.ycombinator.com/item?id=14104156
I was snarky in that thread which certainly didn't help, but you have an uphill battle if you want to implement anything on a large scale.
That is true. But you can observe that a typical CS rep accesses maybe 10 customer's data sets per hour. If they work 8 hours a day, 200 days a year, that's 16k customers they could leak per anno without arousing suspicion Maybe 60k if you allow a fudge factor of 4 to prevent false-positives from highly motivated CS reps.
But only if you monitor for this kind of thing. (And better yet, you should rate limit, and monitor).
Most security problems don't have perfect solutions, but they do have some solutions that get you 80% there. And if you invest in sufficiently many such 80% solutions, you get to a point where it's not economical for an attacker to continue hammering on your application.
It wouldn't have helped in the face of RCE in struts. RCE in struts would let the adversary scp off all the files that make up the database, for instance. This would completely bypass any application-level checks or monitoring.
The main data store that contains everyone's credit reports (high side), should have physical and network isolation from the public site (low side). Only copy from high side to low side when someone becomes a customer.
(disclaimer: IANA security expert, but the perfect is the enemy of the good)
https://www.computer.org/cms/CYBSI/docs/Top-10-Flaws.pdf (PDF warning) might be an interesting start.
The principles below it aren't too hard though: encrypt data in transit and in rest, think hard about key management. Keep attack surface minimal. Validate at all layers where it makes sense (for example in a multi-tenant application, you should authorize access to a row by tenant both in the application layer, and use row-level security in the DB as second layer of defense). Make your assumptions explicit; write them down. If possible, write automated tests that check if they still hold up.
While the principles are not hard to understand, designing an application with them in mind can be quite a different beast, and keeping stuff secure as features and integrations creep in requires quite some dedication.
The attack could be way more sophisticated and involve several hacks through the stack.
Injecting a JSON blob to the Struts REST deserializer that effectively results in executing payload code would indicate high investment in the attack.
The hacker(s) possibly had insider information. They possibly used social hacking techniques to acquire data decryption keys.
Given Equifax stores more than 143M records, the initial report would imply some sort of data segmentation limited the scope of damage. If the doors were wide open, why didn't the hackers take 100% of the records?
None of this excuses Equifax or removes liability (I've personally initiated a credit freeze and will likely need a LifeLock Plus account for the rest of my life thanks to this breach).
Why does it indicate high investment? IMO it's similar to SQL injection attack. People, just take a look at points where complicated data can be deserialized and check whether adequate checks can be overcome.
As I see from their report here: https://cwiki.apache.org/confluence/display/WW/S2-052 they use XStream to deserialize REST data. XStream is basically Java serialization/deserialization to XML instead of binary. As far as I remember from my experience of working with XStream, XStream can deserialize almost any object, even the ones which aren't serializable by default. It's quite trivial to go from here to execution of arbitrary code.
Code that does what? Dumps all environment vars? OK, maybe that produces a DB2 connection string. Refactor code, re-inject, does the connection work? OK. Refactor code, re-inject and dump all tables. etc...
This attack requires an iterative trial-and-error probing process, typically done behind a proxy for many weeks/months.
From Equifax's PR: "the unauthorized access occurred from mid-May through July 2017.".
That's a reasonable amount of time to probe all systems involved and develop sophisticated code.
Of course this attack is quite laborious, but it doesn't require anything special or unique.
For most attackers, it is vastly more efficient to run automated sql/wordpress-vuln/ssh-user scanners dropping bitcoin miners and botnet nodes. Those require zero human effort and self-replicate. That gives you less publicity than a hundred million personal finance records, but it has a much higher success chance and a more reliable income.
If you don't have a full team for this, another strategy is to rely on a vendor, and try to obtain as many dependencies as possible from this vendor. Of course, you still need a process for updating the vendor-supplied components as quickly as possible.
Maybe segmentation is the only possible defence - customers A-C served out of this DB with a different stack to limit the damage.
But once they get in, they get in, right. Its just a matter of finding the next vulnerability.
As long as you are providing a rich customer experience, with the ability to generate reports based on historical information, you have risk.
At some point, the front end of any web application has to go to the back end database to get information. So there's just no way to cleanly and completely isolate the front facing parts of an application from possible breach.
Having said that, definitely keep up on the vulnerabilities of software you use. It's hard though, especially when you're relying on a great deal of dependencies. A company the size of Equifax should have had a team dedicated to this. A team. It doesn't seem like they had anyone who knew anything about basic security at all.