Unfortunately your tax agency is wrong. Finanzamt does not care much. It is the Rentenversicherung that you should care about. They might end up calling Scheinselbstständig. Better get at least one more client to work with.