Re:The DevOps Phenomenon: An executive crash course
by peterwwillis on Wednesday June 05, 2019 @05:56PM
(#20109575)
Not 'nobody on call', just 'unnecessary on call'.
Have you ever gotten paged for a transient error that went away by the time you checked it? Or paged for the same error 5 weeks in a row? These, and really most pages, are fixed by DevOps.
In a DevOps system, a feedback loop is used to address all alerts as bugs to be immediately and permanently fixed. After a while, if something breaks, it's because someone just changed something, so it's happening during working hours. Alerts pop up in slack and are acknowledged before a page is sent out. So nobody is getting called.
If your infrastructure is ephemeral and managed as code, you use CI/CD to deploy all changes, and you aren't resource-constrained (at this point, cloud-native infra is only constrained by budget) you shouldn't have stuff crashing randomly at 3am, so there should be very few pages.
Have you ever gotten paged for a transient error that went away by the time you checked it? Or paged for the same error 5 weeks in a row? These, and really most pages, are fixed by DevOps.
In a DevOps system, a feedback loop is used to address all alerts as bugs to be immediately and permanently fixed. After a while, if something breaks, it's because someone just changed something, so it's happening during working hours. Alerts pop up in slack and are acknowledged before a page is sent out. So nobody is getting called.
If your infrastructure is ephemeral and managed as code, you use CI/CD to deploy all changes, and you aren't resource-constrained (at this point, cloud-native infra is only constrained by budget) you shouldn't have stuff crashing randomly at 3am, so there should be very few pages.