In practice,it's worse than having to use your SSN. You don't need SSN to sign up with all the major apps and sites (including free email),but you do need a phone#.
This sounds like an interesting trade off, but I'm not totally following how using phone numbers and contacts avoids the metadata on server issue.
Unsure how this relates to contact lists though (other than that secret sender is apparently only available for messages from people in your contact list?)
Matrix on the other hand isn't a message-passing system like Signal (or IRC or XMPP or SMTP) - instead it's a way of securely storing your conversations (more like NNTP or IMAP). This means that when you log into a new Matrix client you can get at your conversation history, and it means that even if you lose all your clients you don't lose your history.
The compromise is that Matrix ends up storing the metadata of who spoke to when in the conversation history which is stored on the server.
However, we're working on mitigating this with P2P Matrix (where you run the server clientside, unless you explicitly want to pin that conversation to a serverside server), as per https://fosdem.org/2020/schedule/event/dip_p2p_matrix/ - and it even works :)