Because for example you could have sent them an Email, SMS or called at some point and are in their phone contacts/contact book. That user then shared their phone contacts with FB.

So even though you may not have signed up for FB, they have [your name]{your number} in a table somewhere so that when and if you do sign-up it will be easy to link you with people you already have contacted.