This is exactly how I imagine it would work, albeit with a standard provision being made by Apple and Google as a starting point, something like a universal federated VOIP (they'll come up with a decent brand name for it obvs). If you're on Android you can get UFVOIP in your Google One subscription, or with Apple I'd assume it would be an iCloud+ feature. And then obviously slick contacts integration at the O/S level so that it's seamless to add a new contact - something like when you want to add someone you click "Add Contact" and it gives you a QR code for them to scan on their device to add you.

This could even generate a new relay e-mail for them in the same process so that a single QR code gives them the name and both a unique-to-them phone number and e-mail address to contact you, and adds it to their contacts automatically.

Sharing of contact details could be done by a side-channel:

1) Someone I already know and trust asks to share my contact details to a third party.

2) I get a prompt on my device asking to confirm I want to connect with this new person.

3) A diffie/hellman exchange sets up a secure channel between myself and this new third party.

4) We both confirm we want to add each other.

5) A unique e-mail address and phone number is generated on each device and sent across the channel to the other party.

6) Both parties have now established a new contact for the other party with contact details completely unique to them.

This would entirely resolve a whole class of issues around data protection/harassment/privacy.

If anyone at any time wishes to rescind contact permission they can just burn that contact link. Sure, you could be contacted via someone else that also knows both of you but there's a massive disincentive to pass along your details without your consent - you'll know who passed along your details because they'll be the same unique details you issued to someone else, and you can freely burn their contact link too.