You are thinking from the point of view of a database administrator, whose job it is to make it easy to uniquely identify an entity.

In security it's about making it as hard as possible to deduce even the slightest of information about a person, increasing the k-anonymity as they say.

For all intents and purposes, that a phone number resolves to only two people, and they're also conveniently located geographically near to each other, and are probably very distinct in a whole range of other things, makes it the perfect key for any security adversary.