I would not say the market is small by any possible stretch of the imagination. Minecraft, a game made by one guy in his spire time, is obviously the definition of an outlier. However, it's relevant to note that it sold over 238 million copies. And that's not even scratching the surface of potential, especially globally. The market for games of all sorts is just crazy huge. For some scale, the PS2 is the best selling console of all time (by a wide margin) at 159 million units shipped, the best selling Call of Duty and Elder Scrolls (Skyrim) games both sold "only" about 30 million units a piece.

I think the real issue is that most games are mostly just uninspired knock offs of other games. And this includes AAA games. And here marketing is absolutely critical, because your game may be fun but it doesn't really stand out on its own in any meaningful way. By contrast games that have distinct identities (and, critically, are also fun!) tend to have extremely long tailed success where word of mouth is vastly more important than marketing. An amusing example is Kenshi [1]. The guy went from tens of people playing his game while working as a security guard to make ends meet, to making millions from the game.

And I don't think those examples are particularly rare either, certainly nothing like winning the lottery. But coming up with an idea that's both unique and fun isn't easy.

[1] - https://steamcharts.com/app/233860