The only threat that comes from Signal using phone numbers is that if the police arrests someone you are communicating with, and police somehow unlocks their phone (https://xkcd.com/538/), then they can see what you and them wrote as well as your phone number and therefore know who you are. I agree that in some situations, for example anti-government activists, journalists communicating with whistleblowers or criminals, this is bad. For these situations, Signal is not the solution.