There's nothing inherent in phone numbers here. Both iOS and Android also allows you to add e-mail addresses (and other identifiers) to your local contacts. I'm yet to hear an argument as to why e-mail addresses or other identifiers can't be used in addition to phone numbers, or why it would be a complicating factor.