It's not always them knowing your contact list. Sometimes (most times) this will be powered from the other side, as in someone you know has uploaded their contact list and it's matched you as you registered