A common way to implement this is through SSSD which can cache keys locally when the LDAP server is not responding.

https://access.redhat.com/documentation/en-us/red_hat_enterp...

If LDAP is down in an LDAP environment, you cannot authenticate anyway.

Our systems people get local accounts on machines for disasters like LDAP-down. Everyone else is LDAP-only, including ssh keys.

This is nowhere close to "where we started". Now we have a handful of privileged accounts and centralized auth management across thousands of other accounts.

Centralized logging and centralized auth are pretty much mandatory above some size. Without them, you literally do not know who is doing what.

A way to work around this is to normally work with short term certificate (24h) and allow your systems people to generate longer term certificates (one month) that are stored encrypted on their laptop or usb key.

That way you get both emergency access in case LDAP is broken, as well as a way to make sure old personal access gets revoked after one month.

Yep. I'm a big fan of Hashicorp Vault; that's the next step.

There are of course also DR considerations for that, as always, it is turtles all the way down.

You can have backdoor keys for the root account that you configure during provisioning. The use of these keys/account would trigger a security alert and only be for break glass scenarios. Other situations would use LDAP stored keys for authn/authz and LDAP stored sudo rules for additional authz.

Well absolutely there has to be an alternative way in, be it serial console or a dedicated admin user who does not have keys in ldap. With sshd you would specify a local key instead of ldap for that specific user, for example

Where I work if LDAP (AD) is down, the world stops. In 16 years AD "got stupid" once. So this is a good solution for us.