No, you're not over-reacting. I refuse to give access to my contacts to any app; however, that doesn't stop my friend, mom, or a cousin from giving access and Facebook/Google building a profile on you. The other issue here is Apple and Google have not updated these permissions to only get access to certain contacts you choose, its either all or nothing.