Anonymity is a kind of security. There are other kinds that are trivially not anonymous but should be secured (and private between members), like talking to my family members about our private business.

To the best of my knowledge, Signal's use of phone numbers does not meaningfully compromise the security or privacy of my conversations with my friends or family.