The attention span of players today is much smaller than it used to be and there is a lot of money to be made in this industry, so I believe that these facts changed what mainstream games are supposed to be. But we have the indies :)
You might like this podcast episode where Brian Reynolds talks about the creation of Alpha Centauri:
https://www.idlethumbs.net/3ma/episodes/the-alpha-centauri-s...
My all time favourite game is Alpha Centauri and I would have loved to be around to see how they designed the tech tree among other things. It still feels like an excellent sci-fi novel wrapped as a game. Wish newer games could attempt something like that. Although I have to say many times an RPG game attempts something like that, the quality of their story leaves a lot to hope for.
Darkest Dungeon is one of the newer games that has successfully created great atmosphere, similar to those older games. Yet sadly I at least got very quickly tired playing it, with dull grinding of the characters who would keep dying by random crits.
I guess also because multiplayer has become so prevalent that it's not even necessary to create great storylines anymore. People will be satisfied by the interactions they have playing other humans and its never-ending challenge by the changing metagame and such.