News for hackers. Stuff that matters.
Re:A Simple AI Capable of Basic Reading Comprehension by logicallee on Friday September 18, 2015 @09:32AM (#10239056)
I was intrigued by the initial transcript, but disappointed that it was edited (lightly). For the question "Why did Mary cheer?" the screenshot showed that the simple AI literally answered "because IT be HER FIRST TIME WINNING". This is correct, but the author edited it into a correct sentence for us.

I think it is too much to call this "capable of basic reading comprehension". Surely, "simple sentence parser can answer reading comprehension questions" would be more correct?

I mean it is quite an accomplishment, but there is no understanding here. In some natural languages with less inflection or change in word order, for example, you could answer any "why" questions with the regex /($question_string) because (.+?)\./ against some source corpus, and then $2 will contain your answer. It doesn't work with English due to slight changes in word order, but surely it would be too much to state that this regex is capable of basic reading comprehension in languages it does work in.... If I'm allowed to massage the question the way the author massaged the output, look at this fine result:

http://ideone.com/84QSHC (output at bottom)

Would you say that 14-line Perl program is capable of basic reading comprehension? I wouldn't!

Re:A Simple AI Capable of Basic Reading Comprehension by youngprogrammer on Friday September 18, 2015 @11:13AM (#10239636)
> I was intrigued by the initial transcript, but disappointed that it was edited (lightly). For the question "Why did Mary cheer?" the screenshot showed that the simple AI literally answered "because IT be HER FIRST TIME WINNING". This is correct, but the author edited it into a correct sentence for us.

For some reason, the library I used to get the present tense of "was" is "be". I had a manually fix for this but accidentally removed it when I was cleaning up the code. Sorry if I disappointed you

> I think it is too much to call this "capable of basic reading comprehension". Surely, "simple sentence parser can answer reading comprehension questions" would be more correct?

I would say it is capable of basic reading comprehension because it is attempts to build relationships between different objects although the relationships are weak. When humans do reading comprehension, we do what my program does which is try to parse the sentence and understand the relationships. But brains are also able to augment a lot more information and thus be more flexible with understanding.

> Would you say that 14-line Perl program is capable of basic reading comprehension? I wouldn't!

I would say it is not because it does not understanding the relation between objects; it only understands that sentences starting with "why" should be answered with everything after the string "because". Also, in your program, if there are two instances of "because" in the source, your program will only choose the first one.

Re:A Simple AI Capable of Basic Reading Comprehension by kajecounterhack on Friday September 18, 2015 @11:52AM (#10239903)

  > Would you say that 14-line Perl program is capable of
  basic reading comprehension? I wouldn't!

  I would say it is not because it does not understanding the 
  relation between objects; it only understands that sentences   
  starting with "why" should be answered with everything after 
  the string "because". Also, in your program, if there are 
  two instances of "because" in the source, your program will 
  only choose the first one.
This line is arbitrary. These programs aren't "understanding" anything, they are just able to generate different facts. Your "AI" is very input-dependent too, which was his point.

I am still bothered by the title, as your program doesn't "comprehend" the text -- only you do that. Your program restructures the text and answers to a very limited querying mechanism; there's no comprehension, only information retrieval. They're not the same. Knowledge Graph and Freebase are comparable (/ much larger) efforts.

Re:A Simple AI Capable of Basic Reading Comprehension by logicallee on Friday September 18, 2015 @11:56AM (#10239924)
Thanks for the reply! Thanks for sharing your work with us as well. It's impressive.

It's not disappointing, it's quite a feat. I suppose I would say that what I was trying to say is that "parsing", while impressive, is not a large part of "understanding" in my opinion. To use an analogy: almost by definition, compilers parse languages like C++ much better than humans do (basically perfectly, unless there's literally a bug in the compiler or it doesn't follow the standard due to some error).

But that doesn't mean they understand the programs (at all.) A compiler has no idea on an algorithmic level what a program might be doing (and if you remove comments, maybe a person won't understand it either, if they're not familiar with the algorithm.)

So my basic objection is that you're really calling this reading comprehension, but I don't think anything is actually being "understood"; just parsed. A better title would be as I suggested: simple AI correctly answers reading comprehension questions.

The reason that I object to "comprehension" is that these days there really are a few "deep learning" systems, that can possibly synthesize information. (I don't know that much about them.) I don't think it's fair to elevate semantic parsing to the level of comprehension.

However the comment by tariqali34 makes a good point, that perhaps this is a criticism of reading comprehension tests. I know in multiple-choice tests from standardized exams, I've been able to correctly answer reading comprehension questions about texts that I didn't even read (by finding just the sentence that talks about it), or in other cases, texts that were too technical and that I didn't understand.

I would say that I would be able to answer a question about some biomedical excerpt that I can't understand a word of, I just can't make heads or tails of it, let's say:

In order to study the physiological roles of AGC kinases, a commonly used approach has been to over-express the active forms in cells. However, due to the overlapping substrate specificities of many AGC kinases, it is likely that the over-expression of one member of this kinase subfamily will result in the phosphorylation of substrates that are normally phosphorylated by another AGC kinase. Another strategy has been to over-express catalytically inactive ‘dominant negative’ mutants of AGC kinases in cells. However, such mutants are likely to interact with and inhibit the upstream protein kinase(s) that they are is activated by, and thus prevent the ‘upstream’ kinase(s) from phosphorylation of other cellular substrates. For example, a dominant negative RSK may interact with ERK1/ERK2 preventing the activation of MSK isoforms and hence the phosphorylation of CREB (cAMP-response-element-binding protein) [9]. Furthermore, in Saccharomyces cerevisiae, over-expression of catalytically inactive Rck2p, a kinase that binds to and is activated by the Hog1P MAPK, sequestered the substrate-docking site of the Hog1P kinase, thereby preventing Hog1P from interacting with other substrates. Thus catalytically inactive Rck2P is acting as a dominant negative mutant of Hog1P and not Rck2P.

I couldn't answer a REAL reading-comprehension test about this: I just have no idea what it's REALLY talking about, I don't actually understand it. (Obviously on a syntactic level, it's not hard to parse.) I don't know what over-expression is, I don't know what a kinase is, I don't know what phosphorylation is. I don't understand the text. But if the questions are simple, perhaps I could answer some reading comprehension questions about this by parroting back quotations from it. Syntactically, there's nothing difficult here. I just don't understand it.

So I think it's unfair to call sentence parsing real reading comprehension, even if sometimes reading comprehension tests fail to differentiate between the two. You can parse sentences perfectly while understanding nothing. For example, I could answer the question "what is wrong with studying the physiological role of AGC kinases by overexpressing the active forms in cells?" which the first sentence refers to. I can just quote the second sentence "Due to the overlapping substrate specificities of many AGC kinases, it is likely that the over-expression of one member of this kinase subfamily will result in the phosphorylation of substrates that are normally phosphorylated by another AGC kinase". I don't understand, but I think I correctly parroted.

So there are real issues in determining comprehension. The higher the level of the question that is asked, the harder it is to answer without actually understanding the text.

I did find your work very interesting, thank you.

Re:A Simple AI Capable of Basic Reading Comprehension by youngprogrammer on Friday September 18, 2015 @03:27PM (#10241507)
> So my basic objection is that you're really calling this reading comprehension, but I don't think anything is actually being "understood"; just parsed. A better title would be as I suggested: simple AI correctly answers reading comprehension questions.

I would argue that my program can understand the relationship between different objects but I agree that it does not understand the meaning of the relationships.

> I couldn't answer a REAL reading-comprehension test about this: I just have no idea what it's REALLY talking about, I don't actually understand it. (Obviously on a syntactic level, it's not hard to parse.) I don't know what over-expression is, I don't know what a kinase is, I don't know what phosphorylation is. I don't understand the text. But if the questions are simple, perhaps I could answer some reading comprehension questions about this by parroting back quotations from it. Syntactically, there's nothing difficult here. I just don't understand it.

I would also argue that you are doing very basic reading comprehension here. You might not know the meaning of individual objects, but you understand that "what is wrong" is "the overlapping substrate specificities of many AGC kinases, it is likely that the over-expression of one member of this kinase subfamily will result in the phosphorylation of substrates that are normally phosphorylated by another AGC kinase". You might not know what that whole phrase means, but you understand that its related to "whats wrong" with "studying the physiological role of AGC kinases by overexpressing the active forms in cells". I agree that my program is unable to do full comprehension in not understanding the meaning of objects and relationships, but it can do very basic comprehension in understanding what the relationships are.

> So I think it's unfair to call sentence parsing real reading comprehension, even if sometimes reading comprehension tests fail to differentiate between the two. You can parse sentences perfectly while understanding nothing.

I did not really call it "real" reading comprehension, but "basic" reading comprehension. But I suppose "basic reading comprehension" is still a little of a stretch. I think the real question here is: how can you really determine if a program can understand something? What does understanding something really mean? It is difficult to define something like this and it seems we need some kind of "Turing test" for understanding.

> I did find your work very interesting, thank you.

Thanks!

Re:A Simple AI Capable of Basic Reading Comprehension by tariqali34 on Friday September 18, 2015 @10:08AM (#10239265)
>I think it is too much to call this "capable of basic reading comprehension". Surely, "simple sentence parser can answer reading comprehension questions" would be more correct?

We test whether a human have "basic reading comprehension" by asking them reading comprehension questions. If they answer the questions successfully, then we assume that they have this "reading comprehension" skill. Therefore, if an AI can answer these questions, then it must also have "reading comprehension".

Maybe these "reading comprehension questions" don't actually test reading comprehension, just pattern matching and sentence parsing. In which case, we shouldn't be asking these questions to anyone (human or AI). So what we need are new questions.

Re:A Simple AI Capable of Basic Reading Comprehension by kajecounterhack on Friday September 18, 2015 @11:55AM (#10239920)
We test human "reading comprehension" with the assumption that humans have an underlying world model.

This is starkly different from a machine whose only ability is information retrieval -- this program (and more advanced versions e.g Freebase) parse and restructure text to be more retrievable. But there is no comprehension because there is no world model.

You wouldn't call parsing a google search query "reading comprehension."

Re:A Simple AI Capable of Basic Reading Comprehension by tariqali34 on Friday September 18, 2015 @12:11PM (#10240025)
But if a computer is able to do the same thing as a human does (answer a simple question with pattern recognition and sentence parsing), then the internal mechanism of how a computer does it seems like mere trivia/jargon rather than anything fundamentally important to intelligence. A computer may have reading comprehension without having it in exactly the same way that a human does.

But it's true that we do have a underlying world model by which we understand passages. And that world model is pretty essential if we are to answer more complicated questions. So since it's pretty clear computers can handle the "informational retrieval" side of reading, we should now be focused on generating a "world model" based on text.

Re:A Simple AI Capable of Basic Reading Comprehension by kajecounterhack on Friday September 18, 2015 @12:23PM (#10240087)

  the internal mechanism of how a computer does it seems like 
  mere trivia/jargon rather than anything fundamentally 
  important to intelligence. A computer may have reading 
  comprehension without having it in exactly the same way that 
  a human does.
Yes, it is trivia what we decide to call it, and computers may someday have "intelligence" or "sentience" without having it in the same way that we do. However I think mechanically structuring data and building a querying tool over that cannot be considered "intelligence" or "comprehension" in the sense we seem to speak of here.

  since it's pretty clear computers can handle the 
  "informational retrieval" side of reading, we should now be   
  focused on generating a "world model" based on text.
Yes, that's what Google's working on with Knowledge Graph. It's quite good, it's integrated with search already.