Data brokers sell contact lists that advertisers upload to facebook for targeting. Maybe this has changed, but last I checked, you could upload lists of emails, phone numbers, and/or facebook IDs.

All you need is for one of those identifiers (email, phone number, facebook ID) to show up on a list somewhere. Since you say your email is unique and only used for Facebook, it's unlikely that was used for targeting. Your phone number could be acquired from somebody else syncing their contact list to some app or service. Your facebook ID could be acquired from you authenticating to an app or service via facebook, or it could be scraped from public group pages.