News for hackers. Stuff that matters.
Re:Heroku: We’ve Heard Your Feedback by charcircuit on Friday May 06, 2022 @08:50PM (#31291168)
>The Heroku team and their colleagues have worked around the clock, including nights and weekends

Can someone more familiar with an event like this tell me what they are working so hard on? I imagine securing the vulnerable service and resetting various credentials doesn't take that much work.

Re:Heroku: We’ve Heard Your Feedback by jolux on Friday May 06, 2022 @08:56PM (#31291196)
You have to make sure the attackers aren’t still in your network, you have to get them out if they are, you have to fully scope out what they messed with, and restore anything that’s plausibly connected. Even in the best case with all the right monitoring systems in place there is a lot of manual work involved from owners of the various different effected services and just managing the overall response adequately. I would say the amount of effort involved is comparable to managing a novel, ongoing SEV1.
Re:Heroku: We’ve Heard Your Feedback by jonstewart on Friday May 06, 2022 @10:17PM (#31291611)
Additionally there are legal concerns, for both regulatory compliance and for prepping for inevitable litigation. Those concerns aren’t necessarily a blocker for service restoration, but it really depends on the systems involved. If service could be restored by rebooting a system, for example, but that system also has data related to customers’ (and their customers’) PII and it might have been accessed by the attacker, then you need to make sure it’s all properly preserved forensically first, so that you can comply with regulations regarding breach notifications. The forensic analysis could then happen, but it’s definitely a “measure twice, cut once” situation with lots of lawyers involved (they won’t understand the systems, but they’ll make you explain everything so they can make decisions about risk; and they are in charge).

Also, generally, it’s a “fog of war” scenario, where you can have so many unknowns to work through in a compressed time period, and sometimes there’s an active attacker and they get a vote, too.

Re:Heroku: We’ve Heard Your Feedback by adrr on Saturday May 07, 2022 @12:06AM (#31292081)
Wouldn’t you hire an incident response team who are experts in figuring this stuff out?
Re:Heroku: We’ve Heard Your Feedback by jolux on Saturday May 07, 2022 @05:14PM (#31298663)
Depends. Often yes but at Heroku/Salesforce’s scale you really need a lot of security expertise in-house too.
Re:Heroku: We’ve Heard Your Feedback by nixgeek on Friday May 06, 2022 @08:53PM (#31291180)
One quick example: combing through logs which can go back months or years can be expensive in time, especially if those are stored in something like Glacier and need to be pulled back into a system which permits analysis.
Re:Heroku: We’ve Heard Your Feedback by BeefWellington on Saturday May 07, 2022 @12:01AM (#31292066)
Proper incident response involves a lot more than just "securing the vulnerable service and resetting various credentials" and it will in this case probably involve a lot of different teams.

For starters, you have to document everything. At the very minimum the legal team for the company should insist on this, if no other measures, just as a CYA move.

If an attacker had broad access, it's entirely possible an all hands on deck approach is required to help identify (and document) what systems were compromised. Yes, you definitely want a team working on patching the hole ASAP. You also need a team hunting for any possible persistence. Another team probably involving standing up brand new "safe" systems and failing over client systems over to those running the patched software. While that happens yet more people may start doing audits of what was compromised on the original systems.

I've seen incidents where 50-80 people were pulled in to work on an incident at a company of about 150 employees. Depending on how well-funded your SoC is they can cut that number down substantially.

Re:Heroku: We’ve Heard Your Feedback by csours on Friday May 06, 2022 @09:02PM (#31291231)
I imagine they are examining their whole attack surface area. Probably bringing in consultants to discover what could possibly be attacked.
Re:Heroku: We’ve Heard Your Feedback by riverdroid on Friday May 06, 2022 @08:54PM (#31291184)
Besides that, lots of incident response work to figure out the scope and impact of the breach, I'd imagine.
Re:Heroku: We’ve Heard Your Feedback by wly_cdgr on Friday May 06, 2022 @09:56PM (#31291504)
They're working on their optics, amigo