If you're on iOS you always have to opt-in to give an app access to photos or contacts. On Android, you can't do anything about it now, but it should be moving to a similar system with Android 6 (marshmallow). Neither allows you to restrict access to accelerometers, light sensors, etc as far as I am aware.

If they siphon up your contact list and store it before Marshmallow comes out, blocking it later won't do you much good.