You're right, facebook and others can (and do!) infer people based on contact lists, even if you don't have a facebook account, if all your friends do then Facebook already knows who you are, who your friends are, you're likely interests (based off friends), etc.

The best I figure I can do is to keep the apps I used locked down if possible - and to avoid "stupid" apps that clearly have little purpose other then dataharvesting.

Cool Selfy app? Ehh pass.