NSA only knows then that somebody with that phone number is using Signal ... so what?

They would find out anyway b/c your packets are sniffed on a regular basis and so they know your phone is communicating with the Signal server and maybe the packets itself are characteristic.

So in my humble opinion this criticism is irrelevant.