You don't need to have their number, if someone else has both of your numbers and has uploaded their contacts list then you're linked in that third party's list. Equally likely it's linked via the wifi SSID or GPS?