this is absolutely the truth!
I would say that perhaps what is lacking is a clear vision of the product. Steve jobs didn't use metrics for the initial design of the iphone.
I suspect that as a company grows bigger and bigger, the product loses the initial visionary that created the success in the first place. May be the "founder" (of the product, not necessarily of the actual company) left, or moved on, and what remains is just diluted responsibility between a few different people. Therefore, they resort to metrics, as they have no idea what works, nor have a vision.
Time spent scrolling is not time spent watching ads. So that is definitely not a metric YouTube is optimizing for.
Except that sometimes people do care. It's pretty hard to know in advance whether https://xkcd.com/1172/ will apply or not. You can't run any kind of UX study because odds are that the only way you get people to interact with the feature is by making up an artificial scenario that basically forces them to use it. You can't even run an A/B test because every fewer people will actually reduce their YouTube usage over this. They might be unhappy, but their level of happiness doesn't show up in metrics.
I don't know the figure, but if I had to guess, I would wager that 95% of their views at any given time go to videos produced in the most recent month. Video is incredibly expensive to store, and almost all of what they're storing is basically sitting in a graveyard.
My immediate assumption is these videos are not edge cached, so naturally they want their platform to avoid delivering these types of videos. They would rather stick to the well-cached, proven popular videos that are currently circulating.
The option was removed as part of a general UX refresh that moved from a sort drop-down to pill buttons -- this general drop-down -> pill thing seeming to be a general trend on apps and websites now.
But it's hard to imagine why it was removed in the first place. Saving a minuscule amount money by reducing views of less-cached long-tail content doesn't seem important enough. Nor does there seem to be any compelling engagement reason to prioritize newer videos. And I'm not aware of an ad revenue reason, such as older videos being less monetizable.
Maybe it was just a misguided attempt at UX simplification that backfired?