We are talking about attractiveness here, you can't choose who you are attracted to. So in one case you have a wife who fucks the pool boy, in the other you have a wife that wants to fuck the pool boy but doesn't since she is a great person. But in both cases you have a wife who is more sexually attracted to others than to you, you can live with that but it is sad for those people. Just like how old men tend to be more attracted to young women than their old wives, and that makes those wives sad even if those men never leave their wife or cheats, reality has a lot of sad facts like that.

You'll never be the most attractive person in the world though. Even if your SO think you're attractive, there's a huge amount of people they'd be more attracted to, if they had the chance to meet them.