Re:Summary of June 8 outage
by cowsandmilk on Wednesday June 09, 2021 @06:21AM
(#27446276)
> one customer that gets served from 85% of their servers
There could be a feedback loop that is the opposite of a smoke test.
1. Validate customer configuration, if passes, assume it can roll out
2. Roll out customer configuration to node
3. Node goes down
4. Migrate all customers on node to new nodes
5. Node that problematic customer was migrated to goes down
6. Rinse and repeat as problematic customer migrates to every node and takes out every last one.
There could be a feedback loop that is the opposite of a smoke test.
1. Validate customer configuration, if passes, assume it can roll out 2. Roll out customer configuration to node 3. Node goes down 4. Migrate all customers on node to new nodes 5. Node that problematic customer was migrated to goes down 6. Rinse and repeat as problematic customer migrates to every node and takes out every last one.