I'm not sure that's true. The actual bottleneck with these games is not money but time and interest: you have to have a group of people who all like the same game, and you have to be able to get the group together regularly to play it. There are plenty of people who are interested, but I suspect that the vast majority of them are already players of whatever game they favor (if they haven't switched to something like World of Warcraft--see below), so I don't know that there's a lot of untapped market share.
As for getting the group together to play, when all of the players are teenagers in school or college, that's relatively easy. It gets a lot harder when people are adults and have to work, pay mortgages, deal with various curve balls that adult life always throws at you, etc. And if you know you're not going to be able to devote the time to it that you used to as a kid when you learned the games (I started playing D&D in eighth grade, and spent far more hours playing it between then and graduating from high school than I have in all the years since--and my 40 year high school reunion is this year), it gets a lot harder to justify spending money on game materials.
Another factor is the availability of online games like World of Warcraft, which I mentioned above. These games solve the time and interest problem in a way that can't help scaling much better than in person tabletop play: you can play from your computer at home and there are always other people online playing. There are also limitations, of course: for example, you can't make up your own adventures and you can't have house rules, which were both features of every D&D campaign I have ever encountered. But it's another factor that I think tends to limit the size of the available market for games like D&D.
All of the above factors seem to me to indicate that there is a natural limit to the size of the market for games like the ones WoTC is producing, and the actual active market is close to that limiting size. If that's true, then the option of trying to make more money by scaling instead of markup simply isn't available to them.
For campaign based games like DND and other RPGs I'd agree, but with the specific examples of MTG and 40k I highly disagree. You do not need to meet regularly for MTG (card) or 40k (miniature) style games as they can easily be played as one-offs. While RPGs have one-shots they are far less common and the overhead is quite high compared to MTG/40k (just show up with your deck/army). In addition to this, games like MTG and 40k are often played 1v1 (though can expand) and this makes the scheduling problem easier since you only need to find compatible times for 2 people rather than 3-5 (or more).
I was in a similar boat, though for a long time during university I shared a collection with a friend and we did a pretty good job at selling off stuff to keep the effective costs down.
Since 2016, I stopped playing heavily rotational formats like Standard [for those unfamiliar, there's many different formats of MTG which are played and 'Standard' is one of the most popular ones in which you can play "all the mainline cards released from the last X sets", normally ~2 years worth], and moved to Modern [all cards printed since mid-2003 are playable]. My hope was that because of the longer (and growing) availability of cards, Modern would be more affordable in the long term - since decks would remain viable for long periods of time (perhaps with small adjustments with new cars coming out).
Then over ~20 months, that view changed a lot. First Wizards banned Oko & Mox Opal in January 2020. Firstly, these cards lost value directly - but they also caused quite a big shift in the metagame to the point where entire decks got awful, and hence less valuable. Then in mid-2021 they released the Modern Horizons 2 set, which had a massive impact on very many Modern decks (per MTGGoldfish, 8 of the 10 most commonly played creatures in the Modern metagame at the moment are from Modern Horizons 2). Once again, this totally decimated the value of many existing cards.
I hate the sort of "MTG Finance" approach to the MTG scene, but like you say - it's hard to justify $30/hr for a hobby, and its clear that Hasbro will print whatever it takes to maximize their profits (even if it has a significant negative impact on existing customers).
That would seem the biggest way to stick it to WotC: devalue their artificial scarcity while still enjoying the game.
The main limiting factor is what you can convince folks to play. Game stores have the main formats, and unknown amounts of money spent, but you can do whatever with your friends!
I think I may be done, though. The power and complexity creep has gotten to me in the past few years. I can understand that cards need to be rebalanced, or that new rules get introduced over time, but the complexity of the current version of the game has gotten out of hand. The cynic in me says that the power creep is there so that people playing formats like modern will pay for new sets, and since the new cards are more powerful and have more abilities, it drags down the game.
But there is something to be said about games produced by companies that don't rely upon a constant revenue stream (MtG, e.g.) to enjoy.
There's a thriving comunity out there, and so far I've been welcome with open arms every time I've reached out.
We see this happen basically everywhere with almost no exceptions.
I played D&D in high school when it consisted of a few (often xeroxed) rulebooks, some funny looking dice, some graph paper, and the players' own imaginations. The current version sounds awful by comparison.
Now get off my lawn.
One of my favorite movies is a biodoc about a Soviet-era player from Kazakhstan, on youtube (1st of 3 parts, around 15 minutes each: https://www.youtube.com/watch?v=T_Io7jbHsYs ). I can't imagine someone making a movie like that about a Warhammer player. But, that's just me.
Abstract games are highly, highly dependent on memorization and repetition. In lower tiers a Go or Chess player that's played more games or memorized more positions will handily beat a player who hasn't. The sheer hours needed to get to high level chess play, where memorization doesn't matter as much anymore, is insane. There's a reason most great chess players start out young.
Abstracts are also perfect information games. I like games that simulate aspects of real life, and anything with perfect information immediately breaks my immersion.
Added: Look at some of the PogChamps tournaments on youtube. The players are youtube streamers who are chess beginners and they play horribly, but they are still a blast.
Chess is not a great shared game for a group of friends. You all need to be somewhat close in skill to each other (or else the outliers will get bored of playing chess with the group). That also means that you all need to progress at relatively similar rates.
The board games I like to play are thematic, introduce a bit of luck, and are engaging for players of all skill levels. They don’t allow for nearly the same level of mastery that chess does, but that’s exactly what makes them so fun to play as a group.
Besides playing, watching chess videos is interesting too. I like the youtube PowerplayChess channel for updates on top level events and historical games. There are lots of other channels too.
From that perspective, the business is as sustainable as Star Wars or Marvel: as long as this Thor offers a better fantasy than generic Thors, the brand will last (and accumulate the advantages of a deep, lasting history), and therefore will outlast competitors.
I'm fond of model railways, and it's discarded lots of that heritage. You look in a '70s or 80s magazine, and the projects are "here's a new (inflation-adjusted) $50 kit" or "assemble bits and bobs from three different $20 kits following a detailed article". Today, the equivalent model is only sold fully assembled and painted at $200.
The same seems to be true with Games Workshop and I've never understood this. The products from both companies have huge markups (painted cards and pieces of plastic) and I'd assume that they could actually have a far larger market and profit by concentrating on scale rather than treating their products as luxury items. There's a clear reason that a lot of people are printing miniatures (pirating) from both franchises. I wouldn't be surprised if open sourced alternatives to these games soon spike in popularity as more people get fed up.
I think a lot of people have had the following conversation (in either direction)
Alice: Do you play Warhammer 40k?
Bob: No, I play MTG. I'd love to play 40k but I can only afford one addiction.
This just doesn't seem like it makes for a sustainable business practice. Even if we might consider that the total cost of entertainment isn't that extreme (there is low diversity for these games compared to a videogame or sports habit), people have a hard time justifying those prices for playing cards and plastic models because they intuitively understand that there is a huge markup.