On the other hand, you can't really expect the average citizen to properly curate a private key, and a private key also doesn't work for verification purposes.
I think the problem would be easily solved without encryption or keys by using the social security number in combination with a user-selected PIN number.
Any time you apply for credit somewhere, you should have to provide the social and a PIN. There should also be an easy way to generate single-use PIN numbers that can be used when applying for credit.
They already have a lot of the infrastructure for doing this. You can already put a credit freeze on your social security number and protect the credit freeze with a PIN, for example.
Whenever I am applying for credit, I simply "thaw" out my social security number for a couple of days. This works pretty well, but it's a hassle because you have to do it for all three agencies. It also suffers from the problem that my credit could get compromised if I left it thawed out too long.
Do you mean private key? Or am I about to have a TIL moment? Because your public key is, well, public so I wonder what a compromise of that would look like.
This is just a knee-jerk thought and I'm sure it can be improved, but I believe asymmetric keys are the solution.