The government keeps a publicly available list of the public keys of the people in their jurisdiction. Even the government has no need to know the private keys of citizens. In a sense, a person's identity is the public key. You prove who you are to a third party by encrypting a challenge text provided by the third party. The only reason the government needs to be involved at all is to prevent a single person from having multiple identities, but with or without the government keeping track of the public keys, bank fraud is made exponentially more difficult, and has to be done on an one-by-one individual basis, rather than the situation today where a single hack exposes the credentials of millions.