News for hackers. Stuff that matters.
Re:AllenNLP – An open-source NLP research library, built on ... by TekMol on Saturday September 16, 2017 @03:52AM (#15263486)
They state that it "achieved state-of-the-art accuracies on the SQuAD dataset (Wikipedia sentences) in early 2017"

So I would assume it has bean heavily trained?

Re:AllenNLP – An open-source NLP research library, built on ... by yorwba on Saturday September 16, 2017 @05:24AM (#15263760)
If you look at example questions from the dev set[1], you'll realize that they all use the same words as the sentence containing the answer. Additionally, the topics aren't everyday stuff, but something you'd write a Wikipedia article about. So I guess the model just learns to find the sentence most similar to the question and then selects an answer based on a coarse categorization, which fails when it is presented with unseen situations.

Your example works if you rephrase the question to be more similar to the text:

  Paul gives a coin to Joe.
  Whom does Paul give a coin to?
  Answer: Joe
You can cut the question down to "gives to?" or "coin to?", because that's enough to single out the answer. But as soon as you use s̶y̶n̶o̶n̶y̶m̶s̶ (EDIT: related words) that are not recognized (like "receive"), you have no chance of getting a meaningful answer.

[1] https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/

Re:AllenNLP – An open-source NLP research library, built on ... by adrianbg on Saturday September 16, 2017 @01:59PM (#15265738)
The "Who did What" dataset seems much better in this respect:

Passage: Britain’s decision on Thursday to drop extradition proceedings against Gen. Augusto Pinochet and allow him to return to Chile is understandably frustrating ... Jack Straw, the home secretary, said the 84-year-old former dictator’s ability to understand the charges against him and to direct his defense had been seriously impaired by a series of strokes. ... Chile’s president-elect, Ricardo Lagos, has wisely pledged to let justice run its course. But the outgoing government of President Eduardo Frei is pushing a constitutional reform that would allow Pinochet to step down from the Senate and retain parliamentary immunity from prosecution. ...

Question: Sources close to the presidential palace said that Fujimori declined at the last moment to leave the country and instead he will send a high level delegation to the ceremony, at which Chilean President Eduardo Frei will pass the mandate to XXX.

Choices: (1) Augusto Pinochet (2) Jack Straw (3) Ricardo Lagos

https://arxiv.org/abs/1608.05457

Re:AllenNLP – An open-source NLP research library, built on ... by TekMol on Saturday September 16, 2017 @06:52AM (#15264033)
Yes, that might be a more accurate description. It picks the "who" thingy from the most similar context.

With zero further understanding as it seems:

    Paul gives no coin to Marray. Paul gives a coin to Joe.
    Who got something from Paul?
    Anser: Marray

    Paul gives no coin to Marray. Paul gives a coin to Joe.
    Who received a coin?
    Answer: Paul
Re:AllenNLP – An open-source NLP research library, built on ... by rspeer on Saturday September 16, 2017 @11:08AM (#15264957)
Heavily trained on SQuAD questions. There are lots of models out there that are very good at recognizing SQuAD questions, and reverse-engineering the predictable ways that the Turkers who wrote the questions pulled the information out of the paragraph -- allowing them to answer the question without ever understanding it. https://arxiv.org/abs/1707.07328