Without disagreeing with your observation, as a practical matter the biggest influence is how one responds to the allegations. I remember a few years ago it emerged that a US Senator not only patronized prostitutes, but that his preferred sexual fetish supposedly involved wearing diapers (which is a pretty fringe preference, to at least the same degree as those you mention). Embarrassing of course, especially as it was his second prostitution-related scandal in a decade, but he made the usual tearful public apology to his wife on TV followed by a reaffirmation of his religious faith and bingo, forgiven. His party didn't want to lose a Senate seat if he resigned, and he's since been re-elected and is this year planning to run for governor of his home state.