The original application of the entire field of data science or ML is/was actually based on this paradigm of finding "unconscious preferences" (your words) and hidden patterns. How one chooses to deploy the tech should be judged on its own.
On the current trajectory of tool/data abuse where Palantir et al. are leading the way, this is very low on the sinister scale.
Attempting to normalize that by saying "Palantir is worse" does not make it any less manipulative and sinister.
And to be more on topic, Twitter's value as dataset is overstated. Hardly the panacea people make it out to be.
But of course, line must go up, and it's not you personally being negatively affected, so it doesn't matter.
With the advent of AI, startups become solely about marketing, sales, and defensibility.
So most of the capitalist system will become of this nature. Doesn’t seem like such a good system, and inevitably unsustainable.
When you know what someone will buy based on exploiting their unconscious preferences, and you are paid to increase sales, you will do it. Especially if your competitors are doing it too.
And this happens at scale, invisibly. People never see the manipulation.
In any case, it is not useful for most people. It is useful for the people doing the deceiving.