Even then, that's not retroactive. It's your conversations that are privileged, not you. Removing you as the person's attorney doesn't make the conversations you had as their attorney not subject to privilege.