People scaling SSH authorization tend to set up SSH CAs (either directly, or baked into some higher-level management solution). SSH CAs scale down nicely, as well.