Historically, at least, they sold subscriptions.
Hsbc demanded The Telegraph pull negative stories or they would pull their advertising.
All newspapers are full of stories about property "investment" and have a separate segment one a week for paid advertising of property.
Prior to that there were papers which did indeed make their money from subscriptions. But their content was different as well: explicitly ideological and argumentative. The NYT or Wapo idea of neutral journalism was a later development.
Paid search engine that ranks sites based on how often the users click results from that site (and didn't bounce, of course). The fact that it's paid prevents sybil attacks (or, at least, turns a sybil attack into a roundabout way of buying ads).
Of course, at this point, you are now the product even though you paid. But it's a tactic that worked for WoW for ages.
So having a paid search engine does not fully solve the SEO problem. Having no Ads does not takeout the SEO problem of boosting pages to top.
Getting this hypothetical site to get users is the real problem. Same thing with getting users to a Facebook alternative.
There were search engines around but when Google came out with superior search results everyone switched and those search engines quickly vanished. There are search engines in direct competition with Google today. If Google does not provide the best service there's an extremely low switching cost. Bing, Baidu, DuckDuckGo et al would be happy to take your traffic.
Google is not a search company, they are an advertising company. The more searches you make, the more revenue they make. Their goal is to quickly and often get you to search things. As long as you keep using their platform, the more you search the better.
[1] https://austingwalters.com/is-search-solved/