Instead of classes and levels, popular RPG systems of the time had you develop each skill independently. Some options could only be taken at character creation. A common solution was a pool of character creation points that could be used for magic, other special abilities, status and wealth, or additional points in abilities and skills. Combat systems focused on injuries over abstract hit points and were often more deadly than in D&D. Mostly to emphasize the narrative impact of combat.
By the late 90s D&D experienced a major revival with the release of Baldur’s Gate. I believe the popularity of JRPGs among western gamers went a long way to prove that there was a market for these D&D-style games. So it may have been the case, as you said, that game designers and players (in the west) lost interest in these games but it was only a temporary dip.
Even if TSR had disappeared without fanfare the JRPG market had already been established, so this style of gameplay was not in any danger of disappearing.
Class-based systems, on the other hand, at least break up the FOO strategies into a few separate lanes which give players multiple opportunities to try different things.
I think in general, open-ended systems suffer when they present the player with too many options all at once and it’s hard to design an open-ended system which doesn’t do this. When players are overwhelmed with options they tend to go into “analysis paralysis” mode which can be a frustrating enough experience that they prefer to use a FOO strategy over revisiting that zone of complexity and confusion.
Post-TSR D&D might have capitalized on this, but it was inevitable either way. By that point, generations of nascent game devs had been raised on D&D, and by the time it rose from the ashes following 4E (which I say as a 3.5 baby), we had come full circle and people were sincerely looking to recapture the feeling of WoW at the tabletop, just as WoW's whole genre was born from looking to recapture D&D on the computer.