Yeah but other people share their contact list with WhatsApp and they have you in there, with your name and phone number and possibly more info.

I never understood why Apple allows access to the full address book including all Apple-specific settings such as "spouse" and "home address" that are useful within iOS. There should be a minimal permission mode: name and phone number only.