Yea I was thinking something similar, the alignment of the eyes is also most similar to that class as well.

The reason we find the frown to be most important is our understanding that this is a common symbolic representation of an emotion.

If we were to look at it as just an abstract shape, I think we'd come to the same conclusion as the algorithm though.