Years ago, spam emails were sent to random email addresses found on the internet.
Now spam emails are sent to live email addresses harvested from peoples' activities
Today caller ID is spoofed not even with random phone numbers but specifically phone numbers with area codes relevant to you. For me, it's phone numbers of my area code which I have zero contacts in. Or it's phone numbers from my phone's registered address in Houston.
What makes you think it won't be long before fraudsters are able to connect the phone number they're calling with people likely in your contact list by ... looking up leaked contact lists?
I mean, the odds of them hitting that are so close to zero I don't even want to think about it.