I think the point is not that having an affair is “ok” but that it’s within the range of things that not unusually awful people can do. So it’s “ok” as in “Ok, so you had an affair”, not “It’s ok that you had an affair”.