Those handwritten digits that are a scourge today (e.g. 'would any of these methods work on a different set of symbols?' is unasked) came from a competition to develop a commercial zip code reader for the U.S. Postal Service post back in the day of the Apple Newton. He won it!
I was a bagman for text classification data in the early 2000's and his reviews of the results you got using methods of the time (Naive Bayes, Rocchio, Perceptron, SVM) showed a depth of thought and attention to detail which helped me pick and choose tools to make classifiers with fairly predictable performance and development paths.
GPT-3 on the other hand does a good job of spouting nonsense like Peter Thiel and that has something to do with it's emotional appeal. People make fun of it and laugh at the mistakes it makes like those videos where somebody kicks down one of those Boston Robotics dogs: it's just good enough to be an object for those sort of feelings.
Just to demonstrate, one the most common books during period, "Artificial Intelligence: A Modern Approach, 2nd ed" by Norvig, 1080 pages, has less than one (1!) page dedicated to ANNs. I personally think Norvig is an idiot with regards to Artificial Intelligence, and his book (used in 1500 schools in 135 countries and regions) singlehandedly slowed down the progress of AI by a few years, until a new generation of students outgrew this archaic book.
Published in 2002. At that point, ANN research had reached a pretty hard plateau with very few tangible results. Faulting Russel and Norvig for not going into depth about ANNs is kind of like faulting Richard Feynman for not going into depth about quantum computers in the Feynman Lectures.
Also, a lot of the subsequent work and breakthroughs on ANNs has been done at Google under Norvig's leadership as Director of Research.
Norvig had to be pretty clueless to decide ANNs are such a dead-end, that they don't deserve even a chapter in his book, where all around him there are biological living proofs that neural networks are probably a pretty good bet for AI...
(Note: I held the same opinion in the mid 90s when I reviewed his 1st edition and I'm definitely no Feynmann-level. It's just common sense.)
100 years ago you would have been arguing that all around you are living proofs that ornithopters are probably a pretty good bet for artificial flight. You would have been wrong about that too.
All I'm saying is, Norvig should have been more humble and included a chapter or two about ANNs, with all the research accumulated thus far, instead of betting 100% for the symbolic approach. Let the next generation of students learn both approaches and decide for themselves. It's sad that a whole generation of students was taught AI from this archaic book.
No, that is not all you're saying. You opened with this:
"I personally think Norvig is an idiot with regards to Artificial Intelligence,"
Not only did you lob an ad hominem at one of the most respected members of the community simply for making an editorial decision 18 years ago that you happen not to agree with today, you did it from a newly created anonymous HN account, and then you tried to deny it. Your conduct here has been thoroughly dishonorable. You should be ashamed of yourself.
What is shameful was the extreme shunning of the mainstream scientific community to ANNs, in 90's-00's decades, to the point that it was considered career suicide to publish a ANN paper. I believe Yann LeCun has said similar things in the past[0], reminiscing the time when it took him several years(!) to get an ANN paper accepted for publication.
[0] http://yann.lecun.com/ex/pamphlets/publishing-models.html
https://en.wikipedia.org/wiki/Perceptrons_(book)
applied rigorous math (e.g. when computer science was new) to prove that a certain kind of single-layer neural network couldn't solve certain problems. (Can't learn XOR) It is like proving that it takes N log N comparisons to sort N items.
This dampened interest in neural networks for a long time but the "geometrical thinking in hyperdimensional space" is what the field is all about today.
I think the aversion to ANNs during the 90s was more philosophical and aesthetic - ANNs math is indeed "ugly" compared to symbolic logic, bayesian inference, SVM (in the 00's), and many other traditional AI methods.
https://en.wikipedia.org/wiki/History_of_artificial_neural_n...
I've never seen a language model that could create language models. (Never mind the hardware that runs them.)
You're using a very loaded and narrow sense of 'human' here.
GPT3 doesn't know the difference between a given set of characters and the idea/object the characters represent. It can associate "river" and "stream" and "water" but has no understanding beyond that they appear in patterns together. It couldn't possibly make the connection that river and streams are bodies of water, because there is no association with reality.
GPT3 wouldn't even know the difference between human language and characters derived from some random data source.
The only thing it does is identify deeply complex patterns, as long as there are humans around to notify it when it's doing a good job. It's going to be very useful for auto-complete, and jumping in to help users finish repetitive tasks, along with the other stuff ML is good at, but it's simply a GIGO pattern recognition system.
So I think you have it exactly backwards -- there is a dearth of evidence that AGI is even remotely possible. We have known the full anatomy of the C. Elegans ringworm since 1984 -- it's 1mm long and has 300 neurons. There is a foundation dedicated to replicating it's behavior[1], and all they have achieved is complex animation.
Yet if you asked it if rivers and streams are bodies of water it would probably say that they are.
Likewise if I asked you if black holes and neutron stars are both celestial bodies you would say yes... but you've presumably never seen them, only read about them.
Now I think you could argue that you could ultimately tie your knowledge back to some reality, like star ties to the sun and how you've seen the sun, but I haven't been so convinced that we know enough about how the mind works to be sure that the distinction between form and meaning is real.
Assuming there is no "soul" which makes meat life special, I don't see any fundamental problems with building simulated intelligence.
It isn't conclusive evidence however, and larger models may produce significantly more human like results. But from what we know about how gpt-3 works, all the evidence is on the side of it not resembling human intelligence.
Perhaps now, but if history is any indication, when we (as humans) think we have a good grip on how something really works (like human intellect in this example), we've been wrong.
We model the world around us from observation and testing, find our errors, remodel, and improve over time.
Then at some point we find some piece of information that shows us our model was a decent approximation, but fundamentally wrong, and that we need to start from scratch.
If we find that we want to go beyond the moon (and we eventually will), or that the moon is further than we think, we'll again need a different approach.
I always feel like there's a certain beauty and cosmic humor to it.
It makes people think "driverless", which "full self-driving" is not.
What term would you prefer to describe a car that can get you from A to B without your input?
Explain to them also that you think one highly specialized skill (driving) is the same as the sum of all human knowledge.
Without excessive effort, humans don't have any knowledge of how the world actually works. They only appear to have some level of background knowledge, to the extent that this knowledge is present in their faint memories. But this knowledge is very shallow and disconnected from the underlying reality.
For example, all humans have the notion of object permanence, developed by about six months of life. Object permanence is the notion that things don't go away just because you can't see them.
ML systems need to be specifically trained to have object permanence, and GPT-3 almost certainly does not possess it.
Like, I get that it's hip to booster ML and GPT-3, and all of the stuff humans can do seem trivial, but it's really not the case and is something that is holding progress in AI back massively.
My comment was merely a jab at Yann's poor argument. I don't find humans to be trivial at all, but neither do I believe that they are infinitely complex.
The linked Nabla article is fair, albeit I would appreciate more technical details. It seems to be using the API in zero-shot fashion, which is not what one would do to get the most out of it.
There was a video showing the pattern recognition layer of a car 'forget' about objects it had previously correctly identified. There clearly is room for improvement, and some level of concern is warranted, but it isn't as much of a big deal as it seems, because surely at the higher level objects are remembered. To draw an analogy with human cognition, humans also aren't consciously aware of all the objects in their field of view all the time, but that doesn't mean they forget their existence.
> There clearly is room for improvement, and some level of concern is warranted, but it isn't as much of a big deal as it seems, because surely at the higher level objects are remembered.
I would be very surprised if this were true.
> To draw an analogy with human cognition, humans also aren't consciously aware of all the objects in their field of view all the time, but that doesn't mean they forget their existence.
This is exactly object permanence, in that babies start looking in the right direction for objects that disappear. I assume that Waymo at least have some kind of model for this, but am not convinced that it's particularly effective.
Personally, I think that self-driving will require a limited theory of mind model, and we are nowhere near that.
At the very least there must be some kind of a trajectory model which implies that multiple observations of the same object are analyzed as a sequence. How would it work without object permanence? Would it just erase objects undetected in the current frame and pretend they never existed? I suspect you're too focused on the vision aspect of the vehicle control system.
Some people have completely unrealistic expectations about what large-scale language models such as GPT-3 can do.
This simple explanatory study by my friends at Nabla debunks some of those expectations for people who think massive language models can be used in healthcare.
GPT-3 is a language model, which means that you feed it a text and ask it to predict the continuation of the text, one word at a time. GPT-3 doesn't have any knowledge of how the world actually works. It only appears to have some level of background knowledge, to the extent that this knowledge is present in the statistics of text. But this knowledge is very shallow and disconnected from the underlying reality.
As a question-answering system, GPT-3 is not very good. Other approaches that are explicitly built to represent massive amount of knowledge in "neural" associative memories are better at it.
As a dialog system, it's not very good either. Again, other approaches that are explicitly trained to perform to interact with people are better at it.
It's entertaining, and perhaps mildly useful as a creative help. But trying to build intelligent machines by scaling up language models is like a high-altitude airplanes to go to the moon. You might beat altitude records, but going to the moon will require a completely different approach.
It's quite possible that some of the current approaches could be the basis of a good QA system for medical applicatioms. The system could be trained on the entire medical literature and answer questions from physicians. But compiling massive amounts of operational knowledge from text is still very much a research topic.