Related aside: It frustrates me no end that spellcheck still doesn't appear to use any probablistic considerations, like Markov chains, to determine the intended word. And that when I click the next to last letter to make an adjustment it doesn't then change the suggestions to alternate endings, etc.. Perhaps newer devices than I have do this.
For example:
"The city councilmen refused the demonstrators a permit because they advocated violence."
Which party is "they"? There is no lexical information that can possibly answer this question. It depends entirely on an actual understanding of what "city councilmen" and "demonstrators" (in the context of city councilmen and permits!) are, and which one would be more likely to be advocating violence (and in which case that would lead to a permit denial).
Background: Until recently I worked at a symbolic AI company who was tackling this problem. I myself didn't work on this problem directly, but I became 100% convinced that their approach, while a long shot, was the only conceivable way of solving it in a fully generalized way.
https://www.reddit.com/r/NoStupidQuestions/comments/64ae8h/i...
I mean if us humans have a difficulty parsing each other's statements, then why should machines do any better?
It might be hard to come up with examples on the spot, but in everyday life you will routinely come across things you need to refer to by negation which are relatively uncommon.
The most amazing property in my opinion is the fact that it trains itself. (Whereas neural networks are trained by external systems).
I suspect it's related to imprinting. To take the example of filial imprinting, the brain must have some hardcoded notion of what a parent looks like. Then this is used to to build a parent detector, and the hardcoded notion is thereafter discarded. Then the newly learnt parent detector is used in reinforcement learning (near parent = good). Keep in mind that this all happens just after birth or hatching, before the visual centres of the brain have had any chance to train.
Really cool stuff.
Doing just that for 10 years, beating hand-coded systems: https://www-nlp.stanford.edu/pubs/SocherLinNgManning_ICML201... [pdf]
> I would guess that most modern NNs from the NLP area (Transformer or LSTM) would be able to correctly differentiate the meaning.
Yes. See demos like: https://demo.allennlp.org/constituency-parsing/MTczNjYyNA== and https://demo.allennlp.org/dependency-parsing/MTczNjYyNg==
> I think there is no fancy NN (yet) behind Google search,
During the deep learning boom, Google made a huge push towards NN-based NLP. SEO's and their PR calls their efforts collectively RankBrain: https://en.wikipedia.org/wiki/RankBrain
I think we are on the cusp of combining symbolical/logical operations over the vectors produced by Neural Networks (or at least, major effort there). Could be by neatly tying up all these different NN-based NLP modules (parsing, semantic distance, knowledge bases, ...) with another set of decision layers stacked on top.
Chomsky: Statistical analysis of snowflakes falling outside the window may predict the next snowflake, but it will do very little for weather prediction, and nothing for climate analysis.
Norvig: Give us enough data and we will get close enough for all practical purposes.
It's easy to say, isn't it? Unfortunately, sticking the word "just" in there doesn't affect the difficulty. I do it all the time, too.
That said, "meaning" is not statistical.
Are you sure of that? After all, we don't have all the same interpretation for every word in the dictionary. Ask a hundred person in the street - well perhaps not these days - the meaning of the word "meaning", how many different explanations will you get? And will you be able to reduce them all to the same "fundamental" meaning?
To extend on that, you can think of the human brain as just another (powerful) statistical model.