Re:Netflix Chaos Monkey Upgraded
by devonkim on Wednesday October 19, 2016 @12:38PM
(#12745303)
"Devops" has bazillions of meanings, but avoiding (human) ssh to production boxes is a generally sound principle these days because our infrastructures are becoming harder to understand by poking at boxes one or two at a time now even for forensic analysis.
[ Parent
| on HN ]
So logging in to a server to check a logfile (assuming i dont or cant do centralized logging) is considered anti-devops ?
Edit: Sorry responded to wrong parent, sigh.
It's just a matter of scale. If you are at the scale of netflix, vm are probably too complex black boxes, and logs output somewhere else anyway. Plus the problem may involve the interraction of several vm, or the network, or other composants together.
At this scale, you basically have to have centralized logging. When you have thousands of parallel instances of a single application, searching logs box-by-box just isn't practical.
Consider also that if you're elastically scaling EC2 instances and you need logs off an instance that's since been terminated, too late! That disk is gone. So again, you need a central log service.
Parsing logs isn't the only way to troubleshoot.