Larry Summers didn't merely say they were different, he used that to argue that it helped explain what others attribute to discrimination - that's different. (And from doing some googling, it's also apparently not why he was fired, even though it did cause controversy.) My point is that merely pointing out that men and women are different doesn't weaken any of the arguments that people tend to use to justify diversity programs.