The social security cards here use to say "Not for identification." Then the IRS started using it as a Tax ID instead of creating their own (my Tax IDs in Australia and NZ, two places I had work visas for, were public as well).

Most countries have some type of citizen identification number, which is attached to some ID with a photo and/or finger prints. The US does not, and the political climate for the past several decades would probably never allow this. The Real ID act has been seen as a sign of the beast by religious fundamentalist and a basic erosion of rights by libertarians et. al.

Passport numbers can't be used either because not all US citizens/residents have a passport and the numbers change when you renew them. Most parents get a SSN for their child at birth. Even people I know with dual citizenship overseas have them (all except for one, and you don't really need one unless you want to go to America to work .. and then you'll also need to pay an immigration/tax lawyer to go back and reconcile all your taxes).

Here's a great video on it: https://www.youtube.com/watch?v=Erp8IAUouus