Someone mentioned SSH certificates. My understanding is that it's the 'proper' solution, but is more work to set-up. Whether it's worth it will presumably be a question of scale.