"The issue of representation lies at the heart of the debate between the logic-inspired and the neural-network-inspired paradigms for cognition. In the logic-inspired paradigm, an instance of a symbol is something for which the only property is that it is either identical or non-identical to other symbol instances. It has no internal structure that is relevant to its use; and to reason with symbols, they must be bound to the variables in judiciously chosen rules of inference. By contrast, neural networks just use big activity vectors, big weight matrices and scalar non-linearities to perform the type of fast ‘intuitive’ inference that underpins effortless commonsense reasoning."
Pithy version:
"As of 2015, I pity the fool who prefers Modus Ponens over Gradient Descent." - Tomasz Malisiewicz [1]
Superlong version: https://plato.stanford.edu/entries/logic-ai/
[1] https://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pd...
[2] http://www.computervisionblog.com/2015/04/deep-learning-vs-p...
The author of the computer vision blog post doesn't seem to know much about symbolic AI. Some of the comments point this out.
I'm busy on other things at the moment, but I intend to develop a rule-based system some time soon. I can rule out a neural network straight away because there's no data available, the rules are explicit, well documented, and have to be followed, and the system has to justify its reasoning.
That is not to say I wouldn't consider using a neural network for a perception task.
The only way you can get symbols to work is with weights. Follow this to its logical conclusion and I think you'll end up with a system isomorphic with the vector approach, with dimensions representing something like symbols.
The fat cat is sitting on a mat.
And even within the world of the sentence, it only has a vague meaning. What exactly makes the cat fat? Is it neutered or lazy or overfed? Why is it sitting on a mat - is the mat outside a door, is it waiting to be let in? What kind of a cat is it - could it be a big, dangerous cat? The sentence is laden with signifiers and unknowns. Western children are taught to consider sentences like these in the abstract, but it's not a natural way of thinking, because it's not practical in a life lived connected to the world.
Abstract hypotheticals are the hallmark of more disconnected concerns, and we teach our children this early, in part using silly, deliberately vague sentences like these, and discouraging curious questions that might resolve the ambiguities.
An AI system that's designed to handle abstract sentences like these is not one designed to understand human language, because humans don't reason like this unless they're thinking analytically - and even then, they do so blinkered with biases and errors.
or someone who did not grow up using the English language.
Or, you know, you can use a cut-off point, like 18 or so.
Predicate calculus is an entirely parallel system that only has a representation in a strict subset of human language. A reverse mapping is hopeless and misguided.
I feel like this line of thought, that you can box up words with really concrete meanings that you can then reason about with logic systems is a kind of trap for people who've spent too much time in an analytic frame of the world. It's one where the smarter you are, the further down the road you can get without realising it's a dead end. At best, such a system could only augment.
It will totally be fine for 99% of applications. Marginal returns.
Peter Norvig, On Chomsky and the Two Cultures of Statistical Learning, http://norvig.com/chomsky.html
Also, we want something that is automatic. It means easily adjustable to other contexts (and languages), inferring information about neologisms (e.g. semantic meaning of emoji), etc.
If my recollection is correct, Peter Norvig changed his views around the time he joined Google. Google have a particular way of doing things which doesn't include symbolic processing.
It should be possible to infer using a symbolic approach, or more simply just provide a new definition.
I've met some really really brilliant people who've been banging their head against that particular wall since the 1980s. The MIT AI Lab crew, for example, poured untold brainpower into symbolic inference. There was the whole "expert systems" movement. This all failed miserably, in disgrace, because nobody could ever get it to work, and "AI" became a dirty word. Later, there was Cyc, which was hyped on and off throughout the 90s. http://www.cyc.com/ After that, there were people who tried to reason over RDF tuples, which didn't work either: http://www.shirky.com/writings/herecomeseverybody/semantic_s...
This idea pops up every 5 or 10 years and wastes a generation of brainpower. It never works. And let me be clear: There have been some terrifyingly brilliant people who were convinced that it ought to work, and who spent years of their life on it.
Meanwhile, any joker who can code up Bayes theorem or single value decomposition can get some results in a couple of weeks. Probability and statistics get results (as do more advanced numeric techniques). Logic deduction fails. I'm not even sure I could explain why. But I encourage you to think long and deeply on what Norvig has written on this subject. Or just buy Norvig's two AI textbooks (written before and after he discovered the joys of probability, basically), do some exercises, and compare the results you get.
Statistics isn't sufficient for fully understanding natural language (for that, a computer would have to go out and experience the world in the first person). But it is necessary.
The idea of breaking meaning into an inventory of discrete components like this is at least as old as Hjelmslev, I think even Saussure touches on it.
Part of it seems to me to be that you're breaking down words into words. Even if you write it in uppercase and call it a primitive, you don't have to spend too much time on cognitive linguistics to know that there's really nothing primitive about MAN or MONARCH...
See also: Andrej Karpathy, The Unreasonable Effectiveness of Recurrent Neural Networks, http://karpathy.github.io/2015/05/21/rnn-effectiveness/, and try to replicate it with any formal semantics (good luck!).
Additionally, formal systems rarely incorporate for actual language, with some things being technically correct, but sounding weird, or things being incorrect, yet - prevalent (and a root for language evolution). See also: char2char translation (which accommodate for e.g. neologisms or typos).
The original Altavista Babelfish (which used SYSTRAN) used rule-based machine translation. It has been replaced by Bing Translator, but my recollection of the Babelfish was that it was accurate enough to be usable, and better than Google Translate was when it was first released. Google Translate has improved a lot recently. My only problem with it is that it doesn't understand the meaning of the words it's translating.
Neural networks are perfectly suitable for perception, e.g. image recognition. No argument there.
I think that's a fallacy that will haunt AI forever (or, more likely, will be the definitive civil rights struggle ca. March 25, 2035 6:25:45am to March 25, 2035 6:25:48am)
We tend to move the goalpost whenever AI makes advances. Where many people would have considered chess a pretty good measure of at least some aspect of intelligence, it seems like mundane number crunching once you know how it works.
It may be that we really mean consciousness when we say "intelligence", although if we ever find an easy formulation that creates a perfect "illusion" of consciousness, it may end up having some strong effects on people's conception of themselves that I don't necessarily want to witness.
If there are symbols, they're in the dimensions of the vector; but words only probabilistically suggest meaning, they don't categorically denote it.
With a symbolic approach, the right thing is to use a different symbol for each meaning, and disambiguate based on context (which you could identify either statistically or by using rules).
In particular, words are not repositories of meaning. They allude to concepts; new concepts are created and get forgotten on a regular basis. The connection between words and concepts waxes and wanes over time, and even the very timeline of a connection's strength can be used for allusion: using language to represent concepts that can only be coherently mapped by using previously-stronger allusions conveys a sense of being old-fashioned, while the reverse conveys future-thinking. Using allusions that are stronger within a milieu conveys social signalling information about group membership. Etc.
There's no way a human-maintained database is going to capture the subtlety here on anything like a timely basis. There's no universal truth, everyone's map is a little bit different, and the map is changing all the time.
People who speak the same language are able to communicate perfectly well, almost all of the time, across continents and centuries. New words are rare compared to existing vocabulary and often soon disappear from use. New concepts can readily be mapped onto existing vocabulary. People are able to learn other languages and improve their own with the help of dictionaries and grammar books. Things like humour, cryptic crosswords, and social signalling are edge cases. And things like deception are unrelated to language understanding at the semantic level.
We're discussing the best way for computers to understand natural language and communicate with people, and in practice that's either going to use unambiguous language or it's going to need human help or verification.
People who speak the same language are still prone to misunderstandings.
We may be overestimating our success in communication. Heck, I'm not even sure if my understanding of soon equals yours. Is it a century, a few decades, a few years? And why is it so different from the meaning when I use it to answer when lunch will be ready?
The former doesn't exist in human languages (we'd otherwise have gotten rid of the lawyers long ago), and the latter is infeasible. There is a another way.
Wouldn't that mean it was an actual full AI?
There is work being done to merge the formal/structural semantics and distributional semantics. I have been working on that for over a year at my startup (not necessarily having successes, mind).
The benefit of using distributional semantics (word embeddings, etc) is GPU processing and libraries. My logical form parsing library uses dynamic programming and chart parsing with a LOT of tree pruning to parse a simple sentence, while distributed semantics merely require me to multiply 2 matrices/vectors together - something GPUs excel at.
I'm quite happy with a hybrid symbolic/statistical approach, which could be useful for disambiguating words and phrases in context.
I think the trick is to avoid generating the parse tree in the first place.
Today's hardware is pessimized for symbol and list processing. They can't be done on GPUs, and CPUs work better on contiguous data.