The point about AI in IF would be to break that mold. Dialogs can be more challenging, for example you could make a game about interrogating people. Or you could tell NPCs to do certain stuff for you. Or these NPCs could have some advanced planning or decision making.
To make IF more interesting, especially for those outside a very narrow niche, you would need them to be a lot easier to navigate, and offer more challenges that don't look like puzzles and don't hit you with a brain-twister at every turn.
I don't follow the IF world closely (though I often want to set aside the time to get into it), but surely they are also heavily focused on the prose too?
If you treat the text as just an interface to get to the underlying formal game mechanics, then sure, IF is just simple puzzles because there's no hand-eye coordination going on.
But if you treat the text as a fundamental part of the experience, then the puzzles are only a small piece.
By analogy, films are the world's shittiest games because — even though the graphics are nice – the gameplay is obviously pretty meager. The player doesn't get to make any choices at all! But with a film, the graphics are the point.
The one benefit of text games over other types of games is that you can essentially implement whatever world you want, without a trillion Dollar budget for graphics and sounds effects. A holo-deck powered by imagination.
Speaking of games in holo-decks... The killer-app for AI-powered IF may be erotic fiction.
This debate has been going on for decades. (See Photopia.)
The overall storyline is replaced with a repetitive but novel task with increasing difficultly, and various other loosely coupled storylines are grafted onto the task in different ways. The task also replaces the explicit puzzles.
It is as though rather than being told the story, you are viewing vignettes of the story through a very narrow window
I'm not going to go down some half finished path in Bandersnatch when Black Mirror has enough problem with single stories.
If you don't make moon logic puzzles, your puzzles can be solved relatively quickly unless you're very, very good at making fair puzzles which are also legitimately difficult. Therefore, you're going to need more puzzles than an FPS or action game which incidentally has some puzzles in order to achieve the same total game length, and making content like that is legitimately hard.
Similarly, in order to make your games playable more than once, you need branching paths, such that you have different routes for the players to go down each time, and I really shouldn't need to explain the concept of "combinatorial explosion" in this forum. Add to that the expense of things like art assents and voice acting (unless you really think text adventures are going to come back in a big way) and it becomes prohibitively expensive to not railroad everyone down one main plot with, at most, a few minor side-treks which quickly rejoin the main line.