Honestly that's my experience with ML in general. And I know any organization that does a lot of ML has things like backtests and holdouts to supposedly measure impact on key metrics but I've seen too many cases where adding X has a +10% impact in some top line metric and then removing it later (and returning to the initial state) has another 5% increase in that same top line metric.

Maybe we need to revise the famous phrase to: Lies, damned lies, statistics and machine learning.

They must know whether people encouraged to watch more Netflix actually retain Netflix at higher rates. As their prices have gone up I assume this is becoming a bigger and bigger issue.

How does your ML model control for opportunity cost? How do you prove a causation between watch rate and retention?

Do algorithms that encourage viewership actually increase revenue, or retention? What happens when the algorithm is so good I under-sleep, miss work and get fired, eventually losing my ability to pay for Netflix? What if I see this in myself and cancel my subscription, instead of using Netflix sustainably? What happens if I genuinely watch all programming I find interesting and move on to reading books on the same topics?

In other words, if the Netflix catalog is perfectly sorted by relevance to my personal interests, how much of it will I engage with? How far can Netflix content creators and licensors follow me down the rabbit hole?