I would contend that the cost of automation is about the same as the cost of documentation plus the cost of having to manually do the work over an over. It's just a cost borne up front instead of over time. But to your point in the article, you have to have a culture that supports bearing that up front cost.
> Most of the steps we had to perform did not lend themselves to automation, also.
I don't understand how that is possible? Could you give an example of a task that can't be automated?
Most attempts I've seen to automate tasks flounder for this reason.
Also, there's no point automating something that happens once a year - the cost will exceed the benefit. Hence the comments about decided which tasks to automate with a backlog and metrics.
Most anything _could_ be automated, eg 'Phone the customer and ask them to engage their network team', 'look for similar-looking strings', 'tcpdump the port, and see if the output is irregular', but is it really worth the effort to? That's where the backlog came in.
BTW I wrote a whole framework for automation
ianmiell.github.io/shutit/
and it's taught me a lot about the hidden costs of automation...
If everything in your infrastructure is deployed with code, then automation is simply the act of making sure the infrastructure matches the deployment described in the code. It's true that automation is just as difficult if deployments are manual. Then remediation becomes changing the deployment code to fix a previously unknown problem, instead of manually fixing a problem. This then gains you the advantage of the problem being fixed in perpetuity.
So they aren't in contradiction if your remediation and deployment are the same process, because then it by definition is always up to date.
I guess I should add the caveat that deployment should be quick enough to solve problems via redeployment.
Shameless plug for people who are feeling inferior while reading this thread: as it happens, my consulting gigs revolve pretty heavily around doing exactly what 'jedberg has described, and it's super worth getting to that point. ;) Email's in the profile if you're looking to get there.
For instance, if you've got machines at customer sites where you manage the software, commercial filers or appliances where you don't really have a shell and you certainly can't spin up a VM, etc., your machines aren't fungible. You can't just redeploy a machine, and just about every problem on the machine is going to be a new one. You want the runbook so that it enumerates how to stop and start things around a human intervention (and yeah, as much of that as possible should be automated), but you can't automate the whole thing without changing your architecture.
If you have the option of starting with an architecture where you don't have these problems, by all means take that option! Maybe do cloud-hosted SaaS instead of maintaining on-prem software for customers, or use some fancy cloud storage API instead of a physical old-school filer. But people who have the old-school architectures need to keep things running smoothly, too.
OP said automating runbooks, and any fixes to production--so the code becomes the documentation of how to deploy, and what exactly was done to remedy an error. OP did not say automating failure responses.
Which brings up OP's proposed question...
>> I don't understand how that is possible? Could you give an example of a task that can't be automated?
Could you give a concrete example? There's a vague reference in the article about automating responses for encoding errors which sounds interesting. It sounds like the system generates lots of server errors, and it's easier/cheaper to communicate them to customers directly instead of making code fixes.
"Check the logs for service X (they're here <link>) and look for anything related to the issue"
"If the user impact is high, write an update to the status page detailing the impact and an estimated time to recovery"
The value of a runbook is that it can make use of human intelligence in its steps. No-one is arguing that you shouldn't be automating things like "if the CPU usage is > 90%, spin up another instance and configure it".
I have a long missive about how logs are useless and shouldn't be kept, but that's for another time. I'll summarize by saying that if you have to look at logs, then your monitoring has failed you.
> If the user impact is high, write an update to the status page detailing the impact and an estimated time to recovery"
I guess technically that would be a step in a runbook, that's fair. Although in my case that was left to PR to do based on updates to the trouble tickets. :)
> The value of a runbook is that it can make use of human intelligence in its steps
I'd rather human intelligence be spent on triage by reading the results of automated diagnosis and coding up remediation software than on repeating steps in a checklist.
Sure, there are uses for checklists of things to check, but even that should be automated through the ticket system at the very least, which I no longer consider a runbook, but I guess some might still consider that a runbook.
> I have a long missive about how logs are useless
> and shouldn't be kept, but that's for another time.
> I'll summarize by saying that if you have to look
> at logs, then your monitoring has failed you.
I have to say, you have a tendency on HN to chime in from the peanut gallery and be a bit unrelenting and even combative because jedberg does things differently.
That's a fair critique, and thank you for pointing it out. I try to always back up what I say with the reasons for what I say, but sometimes I get lazy or don't have time to write it all out. I too worry about folks who speak in absolutes, although in this case I happen to actually believe it.
The medium isn't always the best way to have a deep technical discussion unfortunately.
How does that work?
If you emit metrics as necessary to a time series database, then you should be able to build alerting based on the time series metrics. Your monitoring systems should be good at building alerts based on a stream of metrics and visualizing the time series data.
Sometimes you might have to look at the visualizations to find something, but ideally you then set up an alert on the thing you looked at so you have the alert for the next time it happens. A great monitoring system lets you turn graphs into alerts right in the interface, so if you're looking at a useful graph you can make an alert out of it.
Sometimes logs can be useful, but only after your monitoring system has told you which system is not behaving, and then you can turn on logs for that system until you've solved the problem, but you shouldn't need access to old logs, because if the problem was only in the past, then it's not really a problem anymore, right? If you have an ongoing problem, then maybe have the logs on for that service while you're investing that problem, but then turn them off again.
But having a ton of logs always generating and being stored tend to be fairly useless in practice with a good time series database at hand.
Likewise, turning logs on only after you've seen a problem means you miss out on troubleshooting the root cause of it - if there was a spike of badness this morning but you don't have logs for it, you're missing out on diagnostic information that may have protected you from repeats of that spike in future.
I've also had business guys want to analyse things like access logs in ways that they didn't know previously. Logs provide a datastore of historical activity, which in smaller shops is a cheap data lake.
Perhaps the 'no logs' thing works for your setup, but I think it's bad general advice. And your position is not that logs are useless ("turn on logs for that system until you've solved the problem"), but that retaining logs are useless - quite a significant difference between the two.
That's an important distinction, one that I agree with, and I should make clearer.
Logs do have a purpose, but I'm not sure that retaining them does.
Sure, for a very small shop, throw them on a disk, use awk, sed, grep, and perl to look through them, and call it a day. But once you get to the point of "spinning up a cluster of log servers" or something like it, I'd say you're probably better off investing in monitoring instead.
More than once I've run the entire corpus of requests to a system, ever, through a dummy rebuild as a pretty great integration test. It's a powerful SRE tool. Spelunking through all historical data is just icing on that cake, honestly. As the author says, SRE is basically just an information factory; I'd be haaaaaard pressed to agree with you on throwing away a lot of information -- you don't know what you don't know until you want to know it -- and betting all-in on monitoring. Retaining logs is not the hardest problem SRE deals with, either, but SREs turn around and force unrealistic latency requirements on the query side (I see a lot of ELK deploys running into this).
You have to look at it as an Oracle. Oh, great Oracle of a pile of meaningless logs, cook off this map/reduce and tell me an interesting number that I can put in a Keynote for executives. Definitely not dashboarding from logs data. That's an impedance mismatch that Google gets away with because of the nature of their logging.
Nonetheless, splunk is the most expensive software license on the planet. More than Oracle, yes.
Interestingly, we never used a cluster of log servers. I was always skeptical of their utility. It was grep, plus some hand-rolled utility scripts to interrogate. One was a thing of beauty I spent years on, which saved us a ton of time.
A monitoring system has a lower barrier to entry.
http://datadoghq.com/ => will do ALL of that and much more. You can deploy it in a few hours to thousands of hosts, no problem.
Direct competitor: http://signalfx.com/
Have no money to pay for high quality tool? graphite + statsd will do the trick for basic infrastructure. However it's single host, doesn't scale and only basic ugly graphs are supported.
That's what Grafana [0][1] is for -- i.e. creating nicer displays for Graphite.
However it's single host, doesn't scale
It may take some effort, but it can be done, and much of the heavy-lifting seems to have been done and been made available as open-source.
Here's a blog post from Jan. 2017 [2] from a gambling site about scaling Graphite.
And here's a talk [3] from Vladimir Smirnov at Booking.com from Feb. 2017 about scaling Graphite -- their solution is open-source (links in the talk and slides available at the link):
This is our story of the challenges we’ve faced at Booking.com and how we made our Graphite system handle millions of metrics per second.
(And this [4] is an older, but more comprehensive, look at various approaches to scaling Graphite from the Wikimedia people with the pros and cons listed).
[1] https://github.com/grafana/grafana
[2] http://engineering.skybettingandgaming.com/2017/01/13/graphi...
[3] https://fosdem.org/2017/schedule/event/graphite_at_scale/
It can be done but at what costs? Better get a tool that gets the job done out of the box and does it well.
For starters, if you're operating in the cloud, you cannot get servers with FusionIO drives and top notch SSD. That limits your ability to scale vertically.
Similarly, basic infrastructure health is not giving you the same sort of information (what the software is actually doing) that logging does. In order to do time-series monitoring of your software rather than your system, you need to spend time thinking about what metrics you need to track and how you're going to obtain them.
I run both an ELK stack and a Prometheus stack, and I find they're good for different things.
Since you mention Papertrail specifically in the context of costs - Papertrail is actually a bit pricey relative to the competition. For example, compare https://papertrailapp.com/plans to https://sematext.com/logsene/#plans-and-pricing . I think Sematext Logsene is 2-3 times cheaper than Papertrail.
Lastly, I was at a Cloud Native Conference in Berlin last week. A lot of people have the same setup as you - ELK for logs + Prometheus for metrics. We're running Sematext Cloud where we ship both our metrics and our logs, so we can easily switch between metrics and logs much more easily, correlate, and troubleshoot faster. Seems a bit simpler than ELK+Prometheus...
Sumologic, papertrail, logentries (and many more) for cloud logs. Graylog or ELK or Splunk for self hosted logs.
However, logs should never be send to the cloud, they are too sensitive information to outsource. Server metrics + stats are more reasonable.
Agree, stats and logs cover different things. Need both.
Saying that logs should never be sent to the cloud is overly absolute. Some logs should indeed stay behind firewall, but lots of organizations have logs that can be shipped out to services whose features derive all kinds of interesting insights from logs.
Discalimer: I work at Sumo Logic and enjoy it.
Also the cost of the infrastructure to search the logs and view the logs.
> Sometimes logs can be useful, but only after your monitoring system has told you which system is not behaving, and then you can turn on logs for that system until you've solved the problem, but you shouldn't need access to old logs, because if the problem was only in the past, then it's not really a problem anymore, right?
Some things happen rarely, but can still have large impact. E.g. Imagine a once a day job of moving files which fails twice a month, rendering those files inaccessible.
Incident reviews. If something happened that wasn't covered, then it is added as an outcome of the incident review.
> Are there cases where an automated diagnosis could not be made for an incident and if so was manual recourse possible?
For sure. Manual recourse was to dig in and figure it out either with the command line or the monitoring system or whatever else.
> How would you retrain the 'diagnosing app' to handle the new case?
In most cases the "diagnosing app" was a dashboard on the monitoring system, with a set of relevant graphs, so you would add a new graph. There was also a tool that correlated graphs, so you could add a new graph and correlation.
I definitely agree with that, and it's partly a corollary of 'documentation is expensive and requires costly maintenance'.
Run books/checklists are mostly implemented really really badly.
Automation is the ideal, but is costly, and itself requires maintenance.
Most of the steps we had to perform did not lend themselves to automation, also.