Re:Technical Details on Today's Outage
by notepad0x90 on Saturday July 20, 2024 @04:25PM
(#41019521)
In the 3-4 decades of the security industry, testing signature files to see if they trigger a corner case system crash has never been practiced. You and others are proclaiming yourselves to be experts in an area of technology you have no experience in. This was not a software update!!
Then that's 3-4 decades of massive incompetence, isn't it? "Testing before pushing an update" is basic engineering, they have a huge scale so huge responsibility, and they have the money to perform the tests and hire people who aren't entirely stupid. That's gross malpractice.
Re:Technical Details on Today's Outage
by notepad0x90 on Saturday July 20, 2024 @06:11PM
(#41020456)
testing for software, not for content. you test, and fuzz the software that processes the updates, not the content files themselves. it's like a post on HN crashing HN and you claiming HN should have tested each post before allowing it to be displayed. you test code not data, and I dare you to back up any claim that data processed by software should also be tested in the same way. Everyone is suddenly an expert in AV content updates lol.
A config file IS code. And yes, even a post can theoretically break a site (SQL injection, say), so if you're pushing data to a million PCs you'd better be testing it.
Re:Technical Details on Today's Outage
by notepad0x90 on Saturday July 20, 2024 @09:33PM
(#41021678)
You're right, but "testing" could mean anything, you'd need to have the foresight to anticipate the config crashing the program. Is it common to test for that scenario with config files?
I used to work for Microsoft in a team adjacent to the Defender team that worked on signature updates and I know for sure that these were tested before being rolled out - I saw the Azure Devops pipelines they used to do this. If other companies aren't doing this then that's their incompetence but be assured that it's not industry-wide.
Re:Technical Details on Today's Outage
by notepad0x90 on Saturday July 20, 2024 @09:31PM
(#41021673)
I'm not saying they don't test them, I'm saying they don't do code tests, as in unit tests and all that. I have no idea what they do, I'm just speculating here, but if in fact they do no testing at all, then I agree that would be pretty bad.I would think their testing would be for how well it detects things and/or performance impact and I'd expect it to be automated deployment (i.e.: test cases are passing = gets deployed), i guess they don't have "did the system crash" check in their pipelines? In your experience at MS, did they test for system/sensor availability impact?
You're trying to hand-wave away the inexcusable. The outage is a symptom. The problem is the lack of even the most basic testing.
Clearly these files are sent out without even a minimal sanity check. That is a problem, and it's not something that can be hand-waved away.