Such an adversary can ask the phone company whose phone number a certain phone number is, and they will tell him.
No, not necessarily. In fact I'd claim that we should all use Signal so that usage of Signal would not imply any kind of user profile (would not rconstitute any kind of meaningful signal where one could infer what kind of user they are).
I do believe that there's a spectrum of users with a corresponding spectrum of appropriate threat models.
If my own threat model (that I felt I had to adopt) was particularly gnarly, I would (1) use Signal sandboxed / in a VM, tunnelling all traffic through Tor (e.g. Tor listens on socket exposed to VM so that there's no easy way to work around tunnel), and/or (2) if particularly gnarly - would set up pre-shared one time pads with counterparties where possible, and use them to authenticate further, perhaps encrypt further (maybe just encrypting a session key, to save most of OTP) - essentially definitely not deem Signal enough by itself.
Signal correctly focuses on privacy, not on anonymity. The wider your set of claims as to what kinds of properties (from the set of: privacy, anonymity, censorship-circumvention, etc.) you provide, the higher the probability you're going to screw up with one of these, I'd say. Better to combine several tools where possible if your needs require it.
> so they've basically got everbody's real names.
Based on the above as well as on the fact that having phone number does not by itself == real name PII / identity - claiming "so they've basically got everbody's real names" is not just overly dramatic but also patently false and disingenuous. :( I'm sure you did not mean to dissuade people, but c'mon. Generalising in such a way and then adding hyperbole does not help, imho.