There was Gary Hart who had to withdraw in 1976 (IIRC) due to just a rumored affair. I think things would be much different if he had won.

Now, just a "regular" affair is not even news worthy, to be news worthy, somekind of abuse needs to be attached to the activity. And that does not cause any harm to the candidate, especially if a member of the GOP.