Which definitely seems like shortcut 'on to the next thing and ignore QA due diligence.'
Who's ass is responsible when a dev team starts missing coverage in their tests, like happened here?
My experience with separate QA teams was not encouraging. They dramatically slowed down releases while failing to catch bugs. Their capability to automate was poor, so they were doing manual testing that provided little coverage especially compared to our comprehensive submit-blocking unit and integration tests.
Asserts are much easier to forbid by policy.
so... it wasn't tested
Testing all inevitable code paths which have the capacity to bring the platform down as soon as it’s deployed doesn’t seem like an unreasonable ask for a corp operating at that scale.
So like, the requirements are unknown? Or this service isn't critical enough to staff a careful migration?
- All the code has unit tests and integration tests
- Binary and config file changes roll out slowly job by job, region by region, typically over several days. Canary analysis verifies these slow rollouts.
- Even panic rollbacks are done relatively slowly to avoid making the situation worse. For example globally overloading databases with job restarts. A 40m outage is better than a 4 hour outage.
I have no insider knowledge of this incident, but my read of the PM is: The code was tested, but not this edge case. The quota policy config is not rolled out as a config file, but by updating a database. The database was configured for replication which meant the change appeared in all the databases globally within seconds instead of applying job by job, region by region, like a binary or config file change.
I agree on the frustration with null pointers, though if this was a situation the engineers thought was impossible it could have just as likely been an assert() in another language making all the requests fail policy checks as well.
Rewriting a critical service like this in another language seems way higher risk than making sure all policy checks are flag guarded, that all quota policy checks fail open, and that db changes roll out slowly region by region.
Disclaimer: this is all unofficial and my personal opinions.