Separate from my sibling comment about AWS SSM, I also believe that if one cannot know that a Node is sick by the metrics or log egress from it, that's a deployment bug. I'm firmly in the "Cattle" camp, and am getting closer and closer to the "Reverse Uptime" camp - made easier by ASG's newfound "Instance Lifespan" setting to make it basically one-click to get onboard that train

Even as I type all these answers out, I'm super cognizant that there's not one hammer for all nails, and I am for sure guilty of yanking Nodes out of the ASG in order to figure out what the hell has gone wrong with them, but I try very very hard not to place my Nodes in a precarious situation to begin with so that such extreme troubleshooting becomes a minor severity incident and not Situation Normal

If accidentally nuking a single node while debugging causes issues you have bigger problems. Especially if you are running kubernetes any node should be able to fall off the earth at any time without issues.

I agree that you should set a maximum lifetime for a node on the order of a few weeks.

I also agree that you shouldn’t be giving randos access to production infra, but and the end of the day there needs to be some people at the company who have the keys to the kingdom because you don’t know what you don’t know and you need to be able to deal with unexpected faults or outages of the telemetry and logging systems.

I once bootstrapped an entire datacenter with tens of thousands of nodes from an SSH terminal after an abrupt power failure. It turns out infrastructure has lots of circular dependencies and we had to manually break that dependency.

Exactly this. Have heard it referred to as "break glass access". Some form of remote access, be it SSH or otherwise, in case of serious emergency.

Passive metrics/logs won't let you debug all the issues. At some point you either need a system for automatic memory dumps and submitting bpf scripts to live nodes... or you need SSH access to do that.

This "system for automatic dumps" 100 percent uses ssh under the hood. Probably with some eternal sudo administrator key.

Personal ssh access is always better (from a security standpoint) than bot tokens and keys.

There's a thousand ways to do it without SSH. It can be built into the app itself. It can be a special authenticated route to a suid script. It can be built into the current orchestration system. It can be pull-based using the a queue for system monitoring commands. It can be part of the existing monitoring agent. It can be run through AWS SSM. There's really no reason it has to be SSH.

And even got SSH you can have special keys with access authorised to only specific commands, so a service account would be better than personal in that case.

> Separate from my sibling comment about AWS SSM,

This seems like it’s conceding the point since SSM also allows you to run commands on nodes - I use it interchangeably with SSH to have Ansible manage legacy servers. Maybe what you’re trying to say is that it shouldn’t be routine and that there should be more of a review process so it’s not just a random unrestricted shell session? I think that’s less controversial, and especially when combined with some kind of “taint” mode where your access to a server triggers a rebuild after the dust has settled.

Yes, you nailed it with "it shouldn't be routine" and there for sure should be a review process. My primary concern with the audit logs actually isn't security it's lowering the cowboy of the software lifecycle

> combined with some kind of “taint” mode where your access to a server triggers a rebuild after the dust has settled.

Oh, I love that idea: thanks for bringing it to my attention. I'll for sure incorporate that into my process going forward

The first time I heard it was a very simple idea: they had a wrapper for the command which installed SSH keys on an EC2 instance which also set a delete-after tag which CloudCustodian queried.

You don't. you shoot it in the head and get a new one. If you need logging / telemetry bake it into the image.

Are you from techsupport?

Actually not every problem is solved with the "have you tried turning it off and back on again" trick.

No, what we're talking about is (to extend your very condescending tech support analogy) shipping the customer a new PC from the factory, and telling them to throw the old one away because it doesn't matter. It will only start to matter if they have 3 bad PCs in a row, at which time it becomes (a) a demonstrable failure and not just stray neutron rays (b) an incident which will carry a postmortum of how the organization could have prevented that failure for next time

I did start the whole thread by saying "and then I grew up," and not everyone is at the same place in their organizational maturity model. So, if you're happy with the process you have now, keep using it. I was unhappy, so I studied hard, incorporated supporting technology, and lobbied my heart out for change. Without maturity levels we'd all still be using telnet and tar based version control

Once you reach the next maturity level you realize that some problems and bugs are from bad design, and cannot be fixed by restarting the server.

Fixing bad design is an art, not an organizational discipline. (Sadly.)