Basically, it is irrelevant if they "should" or "shouldn't" be harmed, most of them will feel harm once the affair inevitably comes to light. I don't really agree with it, but even if it is irrational to feel harmed over an affair, and even if it is based on self-referential beliefs, people still feel the feelings, and those feelings hurt.
If the cheater knew those feelings would probably happen, and they already promised they would not engage in the behavior, and then they do it anyway...in secret...that is obviously wrong by most people's standards. There are a lot of important "if"s in there. Notice that I did not say that having multiple partners is harmful, or that leaving a partner is wrong, or that it is "always wrong" for someone in a relationship to go off with someone else for awhile. I did not say any of that.
However, when the relationship is a committed relationship, in the most common, Western, traditional sense, it is wrong to have an affair, especially a secret one. If you disagree with that kind of relationship and commitment, that is totally fine, but don't promise that to someone, and if you change your mind be up front about it.
I guess promising to (not) do `X` and not keeping your word makes you untrustworthy but what is the literal harm on someone other than the self-referential belief that breaking promises is bad? To illustrate, let's assume that `X` is something like "I promise to not eat zucchini without you ever again". How is "cheating" by secretly gorging once on zucchini with some friends harmful to the person you give it to, other than their self-induced idea that it is bad?