There is a simplified educational 200 lines python version [2] of it. It claims 96.8% for the WSJ corpus.
What am I missing here?
[1] https://news.ycombinator.com/item?id=8942783
[2] https://spacy.io/blog/part-of-speech-pos-tagger-in-python
I've been watching the line of research in SyntaxNet closely, and have been steadily working on replacing spaCy's averaged perceptron model with a neural network model. This is one of the main differences between spaCy and Parser McParseface.
The key advantage of the neural network is that it lets you take advantage of training on lots and lots more text, in a semi-supervised way. In a linear model, you grow extra parameters when you do this. The neural network stays the same size --- it just gets better. So, you can benefit from reading the whole web into the neural network. This only works a little bit in the linear model, and it makes the resulting model enormous.
Another difference is that spaCy is trained on whole documents, while P. McP. is trained in the standard set-up, using gold pre-processing. I speculate this will reduce the gap between the systems in a more realistic evlauation. Of course, P. McP can do the joint training too if they choose to. I've reached out to see whether they're interested in running the experiment: https://github.com/tensorflow/models/issues/65
Just installed syntaxnet - tests passed in the following setup.
https://gist.github.com/Hendler/61831e411069815ee4ed490f553f...
INFO: Elapsed time: 908.048s, Critical Path: 640.26s
//syntaxnet:arc_standard_transitions_test PASSED in 0.0s
//syntaxnet:beam_reader_ops_test PASSED in 20.9s
//syntaxnet:graph_builder_test PASSED in 16.3s
//syntaxnet:lexicon_builder_test PASSED in 1.8s
//syntaxnet:parser_features_test PASSED in 0.0s
//syntaxnet:parser_trainer_test PASSED in 46.1s
//syntaxnet:reader_ops_test PASSED in 5.7s
//syntaxnet:sentence_features_test PASSED in 0.0s
//syntaxnet:shared_store_test PASSED in 0.5s
//syntaxnet:tagger_transitions_test PASSED in 0.0s
//syntaxnet:text_formats_test PASSED in 1.7s
//util/utf8:unicodetext_unittest PASSED in 0.0s
Some other notes:
Also using Keras with Theano. Before spaCy, StanfordNLP, Freeling, and/or NLTK.
Actually I found an educational 500 lines python parser from the same author as well [1]. It claims an accuracy of 92.7 for the WSJ corpus.
better at parsing the Penn Treebank than the best
natural language parser for English on the Wall
Street Journal
(I worked on a successor project, OntoNotes, that involved additional treebank annotation on broader corpora: https://catalog.ldc.upenn.edu/LDC2013T19)
The point about overfitting is valid, too, which is another reason why this "most accurate such model in the world" claim is obnoxious.
It's also fair to note that their advance is in fractions of percentage points on this specific dataset over models that are 5-10 years older.
So you're referring to this LSTM?
"Andor et al. (2016)* is simply a SyntaxNet model with a larger beam and network. For futher information on the datasets, see that paper under the section "Treebank Union"."
After spending a few months hand coding a NLP parser, am rather intrigued by LSTM. I like the idea of finding coefficients, as opposed to juggling artificial labels.
Cool. I wonder how much how much Human effort (vs machine time) went into tweaking SyntaxNet versus tweaking Parsey M.?
This is an opensourcing of the March 2016 method (syntaxnet, note that in the paper there are results from several trained models) as well as a trained model that is comparable in performance but faster (macparseface).
It is very hard to separate those two things from the way they write.
"On a standard benchmark consisting of randomly drawn English newswire sentences (the 20 year old Penn Treebank), Parsey McParseface recovers individual dependencies between words with over 94% accuracy, beating our own previous state-of-the-art results, which were already better than any previous approach."
From the original paper, "Our model achieves state-of-the-art accuracy on all of these tasks, matching or outperforming LSTMs while being significantly faster. In particular for dependency parsing on the Wall Street Journal we achieve the best-ever published unlabeled attachment score of 94.41%."
This seems like a narrower standard than described, specifically being better at parsing the Penn Treebank than the best natural language parser for English on the Wall Street Journal.
The statistics listed on the project GitHub actually contradict these claims by showing the original March 2016 implementation has higher accuracy than Parsey McParseface.