There are plenty of attacks where some random person can get hold of any phone number for a few seconds. It is not a proxy for identity, as it's available on demand for the exact people trying to impersonate you.