Re:DevOps Students Learn the Value of Uptime With 3 a.m. Calls
by x0x0 on Thursday June 16, 2016 @03:52PM
(#11918424)
I've had success granting myself comp time when I've been up late or worked on the weekend to fix ops issues. Eg if I'm up at 2 or 3, I come in at noon or 1pm, and if it takes more than 2 hours, I just take a full day. The one boss who complained shut up when I told him he had 3 options: (1) deal, (2) remove ops responsibilities, (3) I quit on the spot. It's incredibly necessary to push the costs of sloppy code to be really apparent to management.
Re:DevOps Students Learn the Value of Uptime With 3 a.m. Calls
by lostcolony on Thursday June 16, 2016 @05:17PM
(#11918938)
Well said. I've done the same thing, and fought to make it standard department wide. I'm a dev, I -want- to do the right thing, but sometimes management insists on cutting corners. Fine, but the interest payments on that technical debt should be upon their head.
Re:DevOps Students Learn the Value of Uptime With 3 a.m. Calls
by mancerayder on Thursday June 16, 2016 @04:00PM
(#11918486)
Amen, brother.
While it's normal and expected that the 'Ops' or 'DevOps' or the artist previously known as 'Sys Admin' is expected to be available off-hours for stuff they didn't build (read: emergencies-that-are-not-within-their-purview-to-prevent)... on the flipside, the tendency of many companies to reduce the burden of responsibility of developers and product managers who are often directly responsible for instability in the environment, is prevalent in a way that makes my blood boil.
Going back to this article, it's almost a celebration that DevOps folks have to shoulder that burden.
While it's normal to be on-call, being woken up all the time is a sign of a badly run infrastructure OR release/change-management practices that are rushed and/or feeble.
Everywhere I've worked I've stood up against this tendency, analyzing each issue that causes a page and seeing how it could be prevented. A quarter of the time it's technical: creating redundancy, deep diving into an ongoing issue, doing load tests and capacity management, etc. The rest of the time it's political: oh, the new code caused the memory to be sucked dry from the system, this started precisely after the last release (proof, here's a graph from my Check_MK setup); oh, the devs ran a crap query again on the Hadoop cluster even though we warned them not to do X. etc.
I think if they want to prepare future DevOps students, rather than using PagerDuty as per the article, maybe they should give them shots of liquor and strong beer, their livers could use the preparation. It's an incredibly political role.