I guess the general premise is that any national ID number, SSN or otherwise, is liable to be used in all the wrong ways by untrustworthy institutions both inside and outside the government, damaging privacy and opening you up to exciting forms of fraud against your name (/ ID number); it is often treated as a secret, but it is not a secret at all.

— Unfortunately, with enough computers and enough tracking information, bad actors can often do the same thing without a number, anyway, and you can get different kinds of fraud.