So can widespread monogamy in certain circumstances. In fact, that's pretty much why polygamy usually manifests as polygyny -- in societies with norms favoring men fighting, or engaging in other high-risk activities and protecting women (and children, usually, but while that's important to the motivation, its irrelevant to the effect), after certain types of events, such as large scale (for the community affected) conflict, an exclusive monogamy rule means most women have no mates and prevents a society from rebounding (and makes it more likely that there culture will die out).
This is also probably why in history polygamy loses acceptance as a general norm as the scale of society gets larger and these kind of events affecting a whole society become less common.
It's less than desirable for ugly poor men, but doesn't mean victim. There is no victim involved in not being chosen as a mate by a women. If 5 women choose to marry the rich man, that's a choice between consenting adults and it harms no other party.
No one is victimized by some other people getting married, whether it's because they're gay or because they're polygamists.
While the Old Testament contains many examples of polygamy, all of them are neutral or bad examples. Jesus strongly implies that a man should have one wife and Paul explicitly says so, at least for leaders of the church.
From a practical standpoint, widespread polygamy (which usually means polygyny) can lead to imbalances in marriage opportunity between the rich and poor, men and women, young and old, etc. Banning bigamy is a way to regulate the marriage market, in a way. If tax evasion is harmful to society, polygamy is harmful to ugly, poor men.