Apologies, I was using engagement in a different sense. As a game developer I want to keep people engaged with the game - not just arbitrarily "hooked" but actually immersed and invested in the game I have made.

Which metrics would one track to remain on the ethical path, if any?

I would say: talk to your customers, find out what their needs are and try to make your product better at fulfilling them. Focusing on metrics is going back in the direction of treating your customers as means to an end: like grain to be harvested or sheep to be shorn.

Instead try to think of your customers the way an old barber would think of his clients: as friends with individual needs who trust your experience in a particular problem domain.

Of course, this approach is extremely unlikely to lead to a billion+$ exit. But if your goal is to become a billionaire and you want to be as stridently ethical as possible, I don't know how to help you (and would invite you to further examine your priorities).

It’s unclear to me that highly trained professionals are any better at this than a/b tests. Usually it’s the professionals themselves telling us that they’re better.

I guess it depends on where any individual places the task on the skill/luck or simple/complex scale. If the task (in this case, a design) is very simple — how many users who want to complete their voter registration form are able to — then experts may have an advantage.

OTOH, if it’s a complex space like facebooks news feed, I don’t think experts are the way to go. They simply think that they are.

FWIW, I’ve run hundreds of multivariate tests in my career. Every place I go, the design team is upset by the process, for the same reasons argued here. Every time, I challenge them to come up with 1 winning variation and 3 losers. If the designer-identified “winner” performs best, i pay them $10. If any of the losers wins, I get $10.

I have yet to meet a designer that can, a priori, pick the winning variation better than random chance would suggest.

Expertise is not valuable in an unpredictable context. Complex user facing systems like the news feed are impossible to predict.

I’m channeling Kahneman and Mauboussin here, but their theory explains my experience very well.

Trained experts in short term user tricking into more engagement would be marketers and advertisers. Not usability experts. Usability is not easily measurable with A/B tests, as those are usually not long term enough and not done well enough to make any conclusions.

Note that talking to customers is not easy too. To the point that you need to psychoanalyze all the feedback to understand why they make those comments. Useful feedback is very rare. Customers are just not usability experts, they don't really know what they are talking about.

Probably lots of things that require direct user feedback, like output from focus groups such as customer attitudes towards your product or customer impressions of utility and value.