Except that as humans we invented police, welfare and birth control, so now the hot/rich guy has less responsibilities and it can be just one night stands rather than having at least some concerns about upkeep of the harem/mistresses. And marginal guys no longer have an option of a discarded mistresses who has a child that needs at least some kind of support.
Historically, also, some of those harem women were having fun with men who weren't their husband. This is also common in other species, with at least one species of old-world primate that I can't recall the name of having a sexual structure where the females, who are technically part of a harem of an alpha male, will have sex with other males with the alpha male's tacit knowledge, as long as it isn't in his face.
For some the thrill of the cheating is enough to make the partner seem more attractive than they physically are.