Because a) google is better at indexing a given source, and b) every user shouldn't have to build their own search aggregator across the n sites they have privileged access to; that's the role of a search engine (except for it covering all sources not just a few).

See also Aggregation Theory: https://stratechery.com/2015/aggregation-theory/

There is specific discussion of Google as an example in that article, but the whole article is worth reading.

Maybe there's an opportunity for Google to partner with subscription sites to index their content for users who own a subscription -- but what should Google do when there's a ~0% chance a site contains useful information for a non-subscriber?

If you're running a top-notch search engine, why are you excluding the results I care about most?

Because Google can only index what they can see. As noted in the article, the Googlebot only gets to see the first few paragraphs of WSJ articles, so those pages are less likely to rank on searches.

Indexing more of the content (which would be possible by providing the full content to web crawlers) seems to violate Google's cloaking guideline as well: https://support.google.com/webmasters/answer/66355

It's not excluded. They present less content to Google users and are ranked based on the actual presented content.

If News Corp wants search ads for it's paid service, it can by then like anyone else instead of expecting Google to treat them specially, which is what they are really asking for.