I am not applying any overly austere morals here. I think you would be hard-pressed to successfully argue that an affair in the traditional sense is not an immoral act. If you promise to be "faithful" to someone, and you break that promise and try to hide it, it is immoral based on most people's morals and understanding.
Sure, you might disagree with popular morals, that's fine. However, a relationship involves two people, and both should be valued equally. If one person thinks it is fine to break their promise and cheat in secret, so they do it despite knowing how the other person feels...they are putting themselves first. They are not treating both they and their partner as equals. That is the issue. It has nothing to do with some sort of ideological anti-affair "beliefs." It is simple respect.
I guess it is even technically "fine" to think you deserve more liberty than others, or that you think others' feelings don't matter as much as your desires (I'm not saying you do, I'm speaking generally). If that is the case, just own up to it and people can choose how to react accordingly. If you find partners that align with your views, great, everyone is on the same page, but if not, people deserve the truth. Everything else is up in the air, but they deserve to know. Keeping it from them is like robbing them of the choice to deal with it or leave.
Secondly, still, you continue to make value judgments based on your own personal ethics and assume they're universal truths. Whether there is harm (as you say) truly is debatable. Ending a marriage is breaking your contract, doing something your spouse doesn't like is "harmful". Which ones are ok? Maybe this cheating for many people ends up being a one time thing that the person needed to do to either realize they want to end the marriage or realize they want to change something. Or some other outcome. Regardless, it isn't your or anyone else's job to decide, so please keep your moralizing to yourself.