To most people it is a tax, because you're paying it but not getting any benefits from it (yet). This applies generally to any form of public spending that only benefits some smaller segment of the population.

The same for Medicare; everyone pays the payroll tax but most don't get to use it. We'd have more personal sense of investment in our social programs if they helped people more widely, e.g. everyone of every age can use the healthcare they're being taxed for in countries that have nationalized it.