I built a config management tool, Etcha, that uses short lived JWTs. I extended it to offer a full shell over HTTP using JWTs:

https://etcha.dev/docs/guides/shell-access/

It works well and I can "expose" servers using reverse proxies since the entire shell session is over HTTP using SSE.

“All JWTs are sent with low expirations (5 seconds) to limit replability”

Do you know how many times a few packets can be replayed in 5 seconds?

Sure, but this is all happening over HTTPS (Etcha only listens on HTTPS), it's just an added form of protection/expiration.

I don’t understand why this is more secure than limiting SSH to local network only and doing ‘normal’ ssh hardening.

None of that is required here? Etcha can be exposed on the Internet with a smaller risk profile than SSH:

- Sane, secure defaults

- HTTP-based--no fingerprinting, requires the correct path (which can be another secret), plays nicely with reverse proxies and forwarders (no need for jump boxes)

- Rate limited by default

- Only works with PKI auth

- Clients verify/validate HTTPS certificates, no need for SSHFP records.

There's not one answer to your question, but here's mine: kubelet and AWS SSM (which, to the best of my knowledge will work on non-AWS infra it just needs to be provided creds). Bottlerocket <https://github.com/bottlerocket-os/bottlerocket#setup> comes batteries included with both of those things, and is cheaply provisioned with (ahem) TOML user-data <https://github.com/bottlerocket-os/bottlerocket#description-...>

In that specific case, one can also have "systemd for normal people" via its support for static Pod definitions, so one can run containerized toys on boot even without being a formal member of a kubernetes cluster

AWS SSM provides auditing of what a person might normally type via ssh, and kubelet similarly, just at a different abstraction level. For clarity, I am aware that it's possible via some sshd trickery one could get similar audit and log egress, but I haven't seen one of those in practice whereas kubelet and AWS SSM provide it out of the box

Be careful with SSM, it can provide pretty much unlimited access: https://github.com/Cyberax/gimlet

You can use it to tunnel arbitrary traffic inside your VPC.

IaC, send out logs to Splunk, health checks, slow rollouts, feature flags etc?

Allow SSH in non prod environments and reproduce issue there?

In prod you are aiming for "not broken" rather than "do whatever I want as admin".