A big hero pool you can only pick one of each from is probably easier to balance than something like an RTS where there are a fixed set of units for just a few alternate races and the players typically choose between them once at the start of their career.

You have to consider how every single one of them interacts with all of the others and even less serious imbalances will be present in the games all the times for players that are unable to switch to playing anything else.

I don't think that's true at all, since you have to consider how each team of 5 heroes combines and synergises. That's a far larger part of balance than individual heroes. Players also play roles, but typically play more than a handful of heroes and so those combinations are in reality also very numerous.

On top of that, you then have items which change how heroes can play and, more recently, talent trees which can significantly alter how they play also. Dota's balance is such that a hero can play both a carry role and a support role depending on how the player builds and plays them, and there's a number of examples like that which do get done at the pro level.

That's an incredibly diverse scope to balance.

On the other hand limiting it to one hero of a kind per game helps quite a bit. Overwatch had the problem of very static team compositions early on because they didn't copy that particular feature of MOBAs. I haven't been following that closely but I assume changing that helped quite a bit. Of course they might still have some issues because of the small number of heroes available in the game.

Each composition doesn't have to be perfectly balanced against every other either and the system of picks and bans helps.