News for hackers. Stuff that matters.
Re:Google’s Neural Machine Translation System by d_burfoot on Tuesday September 27, 2016 @02:35PM (#12591834)
Fred Jelinek famously quipped: "Every time I fire a linguist, the performance of the system goes up." Here are two hypotheses that could explain this observation:

1) Linguistic knowledge and theory is not useful for NLP work; instead, people should rely entirely on machine learning techniques.

2) Linguistic knowledge is useful in principle, but the field of linguistics has not yet obtained a sufficiently high-quality theory, and it is better to rely on ML than on low-quality theory.

Almost everyone in NLP adopted hypothesis #1; I believe hypothesis #2 is true.

Re:Google’s Neural Machine Translation System by gradys on Tuesday September 27, 2016 @06:35PM (#12593788)
I'd extend hypothesis #2. We might not have a sufficiently high-quality theory, but we might also not know how to integrate the benefits of the theory we have with the benefits of data-driven approaches to NLP like those described in this paper.

It's clear to me that data-driven techniques are an essential part of the answer to the problem of language understanding. It's infeasible to design a complete set of rules for understanding natural language with all its exceptions and fuzzy relationships. I'd point to Wittgenstein for a stronger philosophical argument on this.

That's not to say that theory has no place in language understanding, but it's not immediately obvious how to combine rule-oriented theory with data-driven ML.

There has been work in this area though, and actually several important, state-of-the-art-moving contributions have been inspired by linguistics. My favorite are recursive neural nets, which benefit from the linguistic insight that language has recursive structure. This paper[1], for example, describes a system that combines a dependency parser with a recurrent neural net (gross oversimplification) to yield a state-of-the-art sentence understanding model.

[1] - http://arxiv.org/abs/1603.06021

Re:Google’s Neural Machine Translation System by j-pb on Wednesday September 28, 2016 @06:45AM (#12596682)
> which benefit from the linguistic insight that language has recursive structure.

Really? You needed an entire _field_, and not just, and by just I mean, in an, but not too, extended, sense, really, trivial observation, to get to that conclusion?

Re:Google’s Neural Machine Translation System by MarkMc2412 on Wednesday September 28, 2016 @02:51AM (#12595884)
Do you have any further detail or reference to Wittgenstein here? Curious to learn more
Re:Google’s Neural Machine Translation System by thegreatestape on Wednesday September 28, 2016 @03:49AM (#12596066)
Not OP, but Wittgenstein covers this in his Philosophical Investigations. In the Tractatus, he tried to create an all-encompassing system for language, but in PI he more or less says it's impossible. It's an amazing read in any case, it has totally changed the way I approach philosophy.
Re:Google’s Neural Machine Translation System by MarkMc2412 on Wednesday September 28, 2016 @03:12PM (#12600400)
Ok, thanks!
Re:Google’s Neural Machine Translation System by sidarape on Tuesday September 27, 2016 @04:56PM (#12593126)
> it is better to rely on ML than on low-quality theory.

Isn't ML (and especially deep learning i.e. neural networks) itself a low-quality theory? From what I understand, nobody really knows why it works so well but uses it anyway.

Re:Google’s Neural Machine Translation System by Isamu on Tuesday September 27, 2016 @05:02PM (#12593180)
In this context, higher-quality == produces results == allows automatic translation with reasonable fidelity.

But I agree, it is a not a very explanitory theory since it is too black-box.

Re:Google’s Neural Machine Translation System by murbard2 on Tuesday September 27, 2016 @07:55PM (#12594335)
I love your work on #2, it's a really interesting avenue, but I wonder if both approaches are close to the point where reasoning about the sentence and its meaning becomes necessary anyway if we want to go beyond the current state of the art.
Re:Google’s Neural Machine Translation System by sushirain on Tuesday September 27, 2016 @04:56PM (#12593116)
The linguistic patterns that you mentioned (parts of speech, etc.) are too simplistic to describe language's idiosyncrasy and they ignore language acquisition.

Machine learning and data-driven methods require models with millions of parameters. Linguistic theories have traditionally not used such models.

Re:Google’s Neural Machine Translation System by kafkaesq on Wednesday September 28, 2016 @06:27PM (#12601855)
Almost everyone in NLP adopted hypothesis #1; I believe hypothesis #2 is true.

"Superior ML performance as a pain signal to be carefully listened to", as it were. I'm in camp #2, also.