Yeah. They also don't carry with them the global context they'd need to correctly interpret these. When I say "Bush spent millions fighting his father's war." (For the record, this is a non-political statement: I'm using a sentence that conveniently explains my point). You and I both immediately gather from the context that "Bush" is not a shrubbery, it's a US president. You then gather that it was Bush Jr. Not H.W. You also gather that the war was the US involvement in Afghanistan and Iraq.

Machines cannot do this in the general sense yet. If you specifically train the machine on data in the domain, and then set it loose on AP stories about the domain they do particularly well, but that's a long ways away from beating human reading comprehension.

It's a fair point, but plenty of humans don't actually have the background to know all that stuff either. If you told a eight-year-old that Bush spent millions fighting his father's war, the kid would know that Bush was a person not a shrubbery, but that's about it. The child could read and understand every word of the sentence, but she'd lack all the same context the computer does.

I think that's beside the point. I could come up with a more general example too...

This. Where there is no ability to summarize or paraphrase there is no comprehension. We need to stop calling fast pattern matching intelligence.

If I'm being charitable, whenever I read headlines like "AI beats humans at reading comprehension", I interpret the headline as part sensational, part tongue in cheek. I am, however, surprised that to this day, people actually still believe in the idea of machine comprehension understood in the most literal sense. The whole idea of mechanizing symbol manipulation is significant precisely because it allows us to simulate computation mechanically without invoking comprehension.

You're really underselling the achievements of MS and Alibaba here. These are machine learning systems that are most likely not using grammar rules and matching, but learning patterns from large amounts of text data. Most of these systems use a word embedding, which can be viewed as a body of knowledge -- it knows how words relate to eachother.