> The ultimate source of information about human preferences is human behavior.

Say what now? Humans like to smoke, eat too much sugar, and behave in various ways that are against their own self-professed preferences...

Yeah, this seems like a justification of the 'stupid consumer' advertising-driven model of human behaviour. Human clicks on ragebait, so it must like ragebait. Until it starts avoiding platforms that send it ragebait, then the human is clearly depressed. Because why else would it avoid it's source of ragebait, which is clearly the only thing it cares about?

I've found in general that as platforms like Youtube and Facebook got more optimised for immediate feedback that's supposedly all about my preferences, they became less pleasant overall user experiences. Is it too much to ask for an AI that at least tries to help humans move towards self-actualization? I'm not saying I'd expect it to work out of the box, but some evidence our long-term interests are actually aligned would be nice.

That’s what people like. If you make the AI know that those things are bad, you’ll be acting as big brother. Do we want the government or mega corp setting preferences?

For example, the American Heart Association promotes a high carb diet to those with heart issues. Sadly, evidence based medicine shows such a diet is bad. If the AI did as the AHA says and promoted a high carb diet, we’d have more heart issues in the country.