I'm not a lawyer, but what I remember from talking with self-employed family members:
There are different parts of German law/government that are affected, retirement is only one of them (and just because one authority noticed/thinks you qualify doesn't automatically mean that the others do).
Tax avoidance (since payroll taxes on your salary weren't paid) is another angle.
For the employer, not paying their part in your social security contributions is similar to not paying you your full salary, and thus a criminal matter as well.
I got some letters about that and they asked me to prove that I was indeed self-employed. They didn't like the argument and concluded that I wasn't. The only thing that actually happened after a while is that I had to pay into the public retirement fund against my will. The "Deutsche Rentenversicherung" are seemingly the only people that care. From what I can tell there were no other obligations/rights that appeared after.