News for hackers. Stuff that matters.
Re:Lessons from 10 Years of Amazon Web Services by autotune on Friday March 11, 2016 @05:18PM (#11269951)
Launch a copy of that server with a new image (which I would hope is under a load balancer), update the sysctl.conf file template with a larger fs.file-max limit in whatever config management system you use for that instance, resume logging under freshly cloned instance.
Re:Lessons from 10 Years of Amazon Web Services by dsp1234 on Friday March 11, 2016 @05:38PM (#11270059)
How would you know to do that at all? The proposed issue is that the file handles are exhausted. Thus logging doesn't actually work, and thus you don't know anything about the problem except 'it's not working'. Your solution presupposed that you have figured out the issue, but the question is how do you find out about the problem at all. Without SSH/debugger/whatever, what is the process to inspect the system to find the problem?
Re:Lessons from 10 Years of Amazon Web Services by devonkim on Friday March 11, 2016 @10:28PM (#11271257)
After dealing with different enterprise systems in AWS for a few years now, I'd say that this is what AWS ELB feature to remove a system out of an ASG as part of its lifecycle exists for. If you aren't using similar horizontal scaling strategies, you have a disaster waiting to happen basically and basically deserve an outage you can't diagnose / RCA well. If you can't ssh in afterwards that means your provisioning for automation / instrumentation and you should stay be able to isolate attention to that part of an instance's lifecycle.
Re:Lessons from 10 Years of Amazon Web Services by autotune on Friday March 11, 2016 @07:33PM (#11270630)
Great question. I'd probably look into something like Sysdig Cloud that can collect that information through its own agent, or one of the tools mentioned by seanp2k2. I do also agree with his point on edge cases though, it should be the goal for the majority of your infrastructure while still allowing for edge cases.