While he was president at Harvard, Summers had a fairly obnoxious demeanor. Maybe that is his demeanor in all situations, and he has had a lot of personal success, but it was ill-suited to that particular job. The president is supposed to schmooze donors and not piss anyone off. That's it. Summers pissed someone off on at least a weekly basis. By the time he made these impolitic remarks, there was a long line of pitchfork-bearing villagers. All it took was one person to complain about him in unimpeachable fashion, and the Fellows started hearing about how terrible he was all the time. One of the Fellows resigned because Larry hadn't resigned quickly enough! In exchange for a loan forgiveness, the rest of the Corporation convinced him it was time.

Also he made a number of errors as president, e.g. the Shleifer thing and the unfortunate changes made by his people at HMC.