I'm mainly talking about 3rd party publishers and specifically the JRPG genre. The N64 suffered from a lack of games compared to PS1. It was difficult to do 3D development back then, developers were just getting the hang of it.
In fact that you bring up Game Boy dev as a thriving platform for 2D content is an important point. Pokemon came out during this era. And a 2D N64 based Pokemon would have sold like hotcakes. Instead they built these half baked 3D Pokemon experiences like Pokemon Snap and Pokemon Stadium, which featured no storyline and no overworld.
Even something resembling Super Mario RPG with prerendered graphics on a isometric view would have been fine. It just wasn't generally done on N64. Everyone wanted to mimic Mario 64. The few RPGs N64 did get, like Quest 64, felt so empty, that it would have been much better to just ship it in 2D.
If textures looked better in FF Tactics vs Ogre Battle 64, it is purely due to storage on CD vs cartridge, a notable 10x plus advantage.
Square used a ton of prerendered graphics in their PS1 games. That is to say, they'd farm out the graphics to super computers and then store them as rasters on disc.
Second is Game Freak proper. They have always valued a small headcount. This has been a problem consistently. They never had the staff to make a console game. They always had the same difficulty with building new ideas. And when they were forced to move to consoles with the Switch, they released their worst games yet, absolutely dispiriting games that probably soured millions of kids on the franchise.
They certainly had the staff to make Pokemon Stadium and Pokemon Snap. Most 2D games in the SNES era only needed like 4 to 7 developers. It would have been fine and trivial even to pair Nintendo game artists with Pokemon developers for a true console Pokemon game. It just wasn't done. It took until the Switch to see a true console Pokemon game.
We don’t know everything about the Pokémon corporate structure in the English-speaking world. Presumably Florent Gorges is researching this as we speak. But the common theory is that when Nintendo invested in Game Freak in the early 90s to bail them out, there must have been a requirement of exclusivity for the franchise in the contract. Game Freak made games for other consoles after Pokémon, and the Pokémon cards had Windows games, but the main RPG series that’s been both only on Nintendo and only done by GF. Last year was the first time another company touched the mainline RPGs: another company did the remake of Pearl and Diamond.
I think that's something that's easy to say in hindsight, but hard to justify at the time.
The switch to 3d gaming was probably the biggest technological leap games ever had. 3d allowed a huge amount of experiences not possible with 2d games. And while early 3d games look incredibly dated now, at the time they were mind blowing and cutting edge. The SNES and Genesis were also too recent to really have any nostalgia, so and thus no real desire for throwback or "retro" titles. Any 2d game on the N64 would probably have felt cheap and dated at the time.
Finally, for Nintendo specifically, they did release 2d titles during the N64 generation. They just released them for the Game Boy (and later, Game Boy Color). There wasn't much sense devoting resources to make a 2d game on the N64 when your 2d-only console is still selling really well.