I think we need to be looking at this from the side of biometrics. There are already privacy laws in some states which protect that kind of information about individuals. It's scary to know that if you collect a lot of what may seem like harmless information about an individual in one place, and analyze it through an algorithm or AI, then you're able to accurately predict a lot of extremely intimate details about a person. Just look at the classic example of Target predicting a teen's pregnancy before she even knew it herself, just by observing her buying habits[1].

Imagine if AI advanced to the point where a human brain could be simulated in a computer. If you collected enough data on a person, you could create a virtual clone with their exact same beliefs, opinions, intelligence, etc. Just imagine how easy it would be to exploit someone if you could simulate how they'd respond to anything with perfect accuracy. You could literally enslave people.

And while that's currently scifi, the amount of money companies like Google and Facebook are pumping into AI research means we could land somewhere pretty close to that eventually. If we're still not protecting peoples' data by then, we're all fucked.

[1]: https://www.forbes.com/sites/kashmirhill/2012/02/16/how-targ...