One example is holiday photos: I am ok with stranger taking photos with me in the background. I am not ok with someone spending a lot of effort going around collecting all the photo with me in the background trying to build a personal profile on me.
There is also a security issue, I trust google to keep my email secure, I trust less google to keep my profile metadata secure as selling it is its main business model.
My point is that they already have the information, so any potential misuse or abuse is possible regardless of what types of parsing they're doing. It might just be inconvenient.