News for hackers. Stuff that matters.
Re:What I Learned Managing Site Reliability for Some of the ... by jedberg on Tuesday April 04, 2017 @01:28PM (#14035175)
> Your arguments contradict themselves - if the infrastructure is changing then there's little point spending the time automating your response to its failure. Most attempts I've seen to automate tasks flounder for this reason.

If everything in your infrastructure is deployed with code, then automation is simply the act of making sure the infrastructure matches the deployment described in the code. It's true that automation is just as difficult if deployments are manual. Then remediation becomes changing the deployment code to fix a previously unknown problem, instead of manually fixing a problem. This then gains you the advantage of the problem being fixed in perpetuity.

So they aren't in contradiction if your remediation and deployment are the same process, because then it by definition is always up to date.

I guess I should add the caveat that deployment should be quick enough to solve problems via redeployment.

Re:What I Learned Managing Site Reliability for Some of the ... by eropple on Tuesday April 04, 2017 @06:15PM (#14037741)
FWIW, I agree with your overall notion of where one should go, but it's worth noting that you're describing an extremely optimistic world for probably 90% to 95% of shops. Which is not to say that it's a bad idea at all to get there, but incremental steps are important and they're what I feel are being described in this article (setting aside that that kind of pervasive automation wasn't really a thing outside of bleeding-edge shops at the time he was describing).

Shameless plug for people who are feeling inferior while reading this thread: as it happens, my consulting gigs revolve pretty heavily around doing exactly what 'jedberg has described, and it's super worth getting to that point. ;) Email's in the profile if you're looking to get there.

Re:What I Learned Managing Site Reliability for Some of the ... by geofft on Tuesday April 04, 2017 @02:05PM (#14035570)
Most of the environments I've worked at have been so far from being deployable with code that it would take you significant time and expense to get there, and in the meantime, you need to keep everything working. You need a reliability engineering team that knows how to triage problems and keep the existing system working while they automate what is feasible to automate. I'd probably agree that this is a separate mindset and skillset from keeping an infrastructure-as-code deployment running, and it's unfortunate that we don't have good terms to differentiate them. A lot of the SRE mindset applies, but not all of it.

For instance, if you've got machines at customer sites where you manage the software, commercial filers or appliances where you don't really have a shell and you certainly can't spin up a VM, etc., your machines aren't fungible. You can't just redeploy a machine, and just about every problem on the machine is going to be a new one. You want the runbook so that it enumerates how to stop and start things around a human intervention (and yeah, as much of that as possible should be automated), but you can't automate the whole thing without changing your architecture.

If you have the option of starting with an architecture where you don't have these problems, by all means take that option! Maybe do cloud-hosted SaaS instead of maintaining on-prem software for customers, or use some fancy cloud storage API instead of a physical old-school filer. But people who have the old-school architectures need to keep things running smoothly, too.