There doesn't have to be a match. Simply the fact that either person either uploads their contacts (either via linking to their email or installing Facebook on their phone) is enough to assume the link.

After all, if you have someone in your contact list (phone or email), you've probably been in touch at least once.

Nefarious? Debatable. But also totally predictable.