It’s convenient to just have a phone number be the piece both sides know. It’s not as magical if you need to also give your AppleID to every single contact manually, and if you have some sort of Phone Number to AppleID lookup service it’d be abused just as badly is what I’m saying here.