I think he would have been removed (or removed himself) as university president sooner or later regardless given his other political blunders, but it might have taken a few years longer. As I mentioned, the comments about sex differences provided a convenient excuse and rallying point for his critics.
I wouldn’t say he was “terrible” at his job. Naïve, undiplomatic, and bull-headed, with good intentions but without enough political skill to persuade his opponents to follow him or enough empathy to understand their objections and moderate his positions.
Disclaimer: I was a first year undergraduate at the time, and my understanding comes from talking to various people at Harvard during and after the controversy, including both critics and supporters of Summers. I wasn’t well enough connected to be the ideal first-hand source about this topic.