When has news content not been about maximizing ad revenue? That's been the model for decades before the internet.
This is similar to how today's high-margin advertising funded companies (e.g. Google, Facebook) can afford to pursue out-there projects with no prospect of making money in the short term. Investigative journalism was a vanity/halo product that usually didn't make money, funded by people overpaying to advertise in the newspaper.
In a case of perfect competition and low barriers to entry, newspapers have to prioritize minimally expensive content as a supporting scaffold for advertising. Newspapers are much closer to this "perfect competition" case now than they were in 1990. (A very few newspapers like the New York Times achieved "escape velocity" to reach a national audience on the strength of their reporting.) Likewise, if Google were up against a dozen other nearly-as-good search engines, neither it nor its competitors would have the luxurious cash surpluses for "moonshot" projects.
The golden ages of AT&T and IBM were also back when they enjoyed monopolistic pricing power. They could afford to do Nobel Prize winning research instead focusing slavishly on cost efficiency. They overcharged everyone, and a bit of that surplus money was reinvested in scientific research and product quality that was higher than what an equilibrated competitive market would converge on.
I wouldn't give AT&T a telephone monopoly again just to bring back Bell Telephone Laboratories. But certain kinds of positive externalities seem to emerge only from businesses enjoying fat margins and high pricing power. One of those positive externalities, in the case of newspapers in the 20th century, was deep reporting.
This has always been true. But “old media” was the same for everyone, so maximizing ad revenue was achieved through capturing the middle of the distribution (engagement is lower, but there’s much more people).
Now we can tailor for every individual, which means maximizing ad revenue is about pandering to the tails of the distribution (where people are super engaged even if the numbers are more limited).
This is spot on. The major problem here is that content is now largely generated based on maximizing ad revenue. News publications developing content aren’t all that different from SEM campaigns. When editors and publishers use article clicks as a performance benchmark, as many do these days, this is bound to happen.