I use signal/whatsapp etc without giving them access to my contacts. I have to type in the phone number (only first time) with whom I want to chat. And that's okay.