I'm not a lawyer, but what I remember from talking with self-employed family members:

There are different parts of German law/government that are affected, retirement is only one of them (and just because one authority noticed/thinks you qualify doesn't automatically mean that the others do).

Tax avoidance (since payroll taxes on your salary weren't paid) is another angle.

For the employer, not paying their part in your social security contributions is similar to not paying you your full salary, and thus a criminal matter as well.