This is even before we talk about how to handle the huge number of people who will lose or delete them.
Tech can't solve this prpblem. Any system that requires a secret won't be.
Most modern phones have secure elements that can generate and store a private key that cannot be extracted through software. Also, it is easy to set up things such that a physical confirmation is required to sign something (this is eg. what some U2F keys or touch ID do).
Of course, you still need a procedure to map public keys to identities and people need to secure their phones in order to prevent someone stealing the phone to make signatures.
But using the secure element of a phone for various forms of authentication is orders of magnitude safer than relying on a credit card or social security number.
The "only" other problem that will remain is that you will need a secure supply-chain, otherwise this will happen:
http://www.zdnet.com/article/id-card-security-spain-is-facin...
https://www.reuters.com/article/us-gemalto-cybercrime/hack-g...
If you do something like this you actually need to be serious about it and establish a rigorous vetting/auditing process -- not just hand the contract to whoever donated the most to your election campaign.
Maybe set-up a 3-year long NIST competition or something, like they do when choosing new crypto standards, and establish the winner this way.
The other side of the equation, allowing services to interface with these cards securely, is already being solved by the FIDO 2.0 spec.
Standards like FIDO and others allow for browsers and websites to utilize these cryptographic functions of smart cards in ways to handle website authentication. The technology exists for smartcards to handle authentication, it is simply a matter of us moving to this technology.
USB smart card readers are fairly cheap. Many business-focused laptops have been available with smart card reader options for many years.
The public keys can be stolen all day long and they're the only part of the equation that needs to be stored anywhere long term. The private keys are just that; truly private, and ideally extremely difficult to steal.
Yes, there will need to be a public service to manage the public keys, and yes this will be able to be compromised in some dangerous ways, but not quite so dangerous as "Whoops, everyone's SSNs are lost, now any attacker can impersonate them because that's all they needed."