I just don't think you can automatically assume ill intent. As someone with depression I've been frequently told to "smile" or "lighten up" and while it's certainly annoying (and occasionally infuriating), I know that the underlying intent was probably positive.
Given an ambiguous statement, people can (and should) judge the statement based on context, and that includes perceived intent of the speaker. Human communication is much more than just words. Context and intent absolutely matter.
IMHO, everyone should be given the benefit of the doubt when dealing with such serious allegations. Or, as they say innocent until proven guilty. I wouldn't want to work in a place, or live in a society that believes otherwise. There simply needs to be a higher burden of proof than "it's sexist because I felt it was sexist".
Can it be considered sexist to say what was said? Sure. Should it be considered sexist by default? I don't think that's been proven.