This cut both ways. Power inbalance has the problem that sex is a extremely common strategy for gaining power in human societies. Predators can take any side on the relationship.

All the more reason not to engage (or attempt to initiate) a sexual or romantic relationship with a subordinate or superior.

Well, the problem is that then all we would have is a closed caste system, that is a very undesirable situation in itself and much worse.

Uh, no.

You know it used to be that men and women didn't work together, right? Like, this idea of being able to date at work has only been around for about 50 years or so, while the human race has been around for thousands. Somehow we figured out how to reproduce in the West without a "closed caste system" just fine. You can too, without dating anyone at your job.

There are plenty of people of different social classes than you who don't work at the same company as you, so I don't think a social norm saying that you shouldn't ask out maybe a few dozen people you work with is a "closed caste system".