It's not about the sex. It's about the judgement shown in cheating with a Chinese spy and a direct report, specifically. It's the lack of trustworthiness shown by someone who is willing to cheat in their marriage (presumably they are then fine lying to their company). And finally it's about exporting gpus to China illegally.

This isn't just a case of some extramarital sex.