I think a lot of it comes down to how these things are trained. You can predict a word, and then easily calculate whether it's right or wrong. So it makes it easy to design loss functions around "word things". It's a lot harder to design loss functions around the complex intricacies of word combinations -- whether or not the network correctly predicts the sentence's meaning is much harder to calculation than whether or not it can predict a certain word. For me, the surprise is that so much sophistication pops out after training purely at the word level. However, I guess it's not surprising to find some limitations to that.

But if we did have a way to calculate loss directly on sentence meaning, then you could train a much more intelligent pooling function. In principle. I'm just not sure how that loss function would look. Maybe question answering tasks are part of the key.