I was thinking it would be super cool if you could use something like https://krypt.co/ and use public key private key.

Sure, you would need that ssh daemon running on the computer, but I bet it could it could be retrofitted to use qr codes or something.