They are a very useful identifier. However they only prove that someone with such a name exists. They are not enough to prove that the given person with that number is the unique person they say they are.

It should be harder for you to impersonate someone else via their SSN than it is to take a stolen card and purchase a $5 coffee.

I agree, but what makes it hard shouldn't be having access to that number, it is verifying the number belongs to the person you are trying to impersonate. There is great need for an impersonal unique id for everyone - there are doesn't of people in the world who share the same name as me. It is possible that some of those people share the same birthday as me as well. For most purposes I want to go by my name, but when you need to be sure you need a unique ID.

The case against using SSN in unintended ways is not the problem, I don't think there is any serious opposition to that idea.

The problem is the case for making a viable replacement for such usages.