CIA director is an easy target, and it's more than possible that Petraeus was targeted for some other reason and the affair was a useful pretext for taking him down. It wouldn't affect a president though. Bill Clinton only became more popular once the Republicans tried to remove him from office over an affair he had with a White House intern.