Google Brain/DeepMind are also pushing some of those ideas. They must be, since they aggressively poach all the top researchers from those labs...
Ng approach is different: he wants a world powered by Deep Learning, so his goal is to make applied deep learning thrive. His strategy to do that: give those data-hungry models even more data, which is completely reasonable.
Those two approaches - fundamental research and applied deep learning - are often referred to as AI, causing much confusion.
Are they talking about basically getting the most out of the current ML type systems?
Since Deep learning is a subset of machine learning, the sentence retains correctness but there are now two equally valid interpretations.
Okay, I do have something relevant to say. I don't see what advantage there is in raw sensory data streams, models can be trained off-line to operate on sensor streams just fine. What we actually want are systems that learn adaptively and on-line. To do that well, they'd need to also be data efficient.
I mentioned this in another comment, but I don't know that we can train these models just fine, a human can get a lot more information out of an audiovisual stream than a rudimentary transcription of recognized speech, objects and/or text.
If we make these models more sophisticated to capture more information (e.g. body language, tone, context), we have to decide how that information is structured and communicated to the "higher level meaning interpretation" stage. No matter what, their output is going to be more rigidly structured and will contain less information than the raw stream. The extent to which the sensory processing model captures the human-recognizable information in the stream is the extent to which you have created an intelligent system.
There is some form of this structuring and reducing happening inside our brains, but we will never get machines to do that if we continue curating structured data sets. We, the researchers, are using our human intelligence to process raw sensory data and put it into a nice format for the AI. They need to be able to do that themselves.
Do we? I'm not convinced. I've joined different sensory inputs before and just glommed together the top of two nets, I didn't have to create any of my own representations.
> but we will never get machines to do that if we continue curating structured data sets
Curated datasets are important though. Want your machine to understand more about the world around it? Then you need high quality inputs in formats you can load in. These data also need to be licensed correctly.
> They need to be able to do that themselves.
We don't chuck kids out into the wilderness and expect them to come back as a useful member of society. We have a huge range of inputs specifically curated to help (from toys and shows to school curricula). Later on with specialisation we pay for extremely carefully selected data, presented in a specific order!
Creating high quality datasets is vital for really anything from niche specialist systems to large general ones.
Or is it just the time correlation that interests you? Because some of these data sets are very likely to indeed be time correlated. Like a video/audio data set for example.
You can feed a million images that are labeled "cat" or "no cat" to one of these systems and it can achieve a human-level proficiency at identifying cats in images. But, it won't be able to do anything other than identify cats, it's far too narrow to be considered intelligent in any way.
If you can feed a series of timestamped photos to a system, basically an unlabeled arbitrary video stream, and you can demonstrate that it formed some notion of what a cat is, that would be very interesting indeed.
With modern data sets and labelling, so much of the problem domain is deeply hard-coded into the system. It doesn't have to learn what letters and words mean and how to identify them in a totally arbitrary* visual or audio stream. It just gets a relatively minute amount of structured data that it has an embedded understanding of what to do with.
Sure there are things like OCR and speech-to-text, but I don't think you could just run your streams through those, there's just so much subtle information loss. In order to make AI that really has a chance of reaching what humans would call intelligence, I firmly believe it has to make meaning out of some kind of raw sensory experience analogous to ours.
*Ok, not totally arbitrary, a human would not learn language from a video of a forest, and that's where the "labelling" comes in, from observing other people using language, but a child can still learn any language from just that, and the labels are often vague, inaccurate, contradictory, complex, abstract, etc. There's no master training set with the right answers, you have to decide for yourself. And humans were also capable of bootstrapping language from nothing. I just don't see modern supervised learning systems ever doing things like that.
> In order to make AI that really has a chance of reaching what humans would call intelligence, I firmly believe it has to make meaning out of some kind of raw sensory experience analogous to ours.
And why must it be able to do this from scratch? Why must it be hampered with the same limitations we have?
Self-Taught Learning: Transfer Learning from Unlabeled Data
http://www.andrewng.org/portfolio/self-taught-learning-trans...
The problem with the dataset-first approach is that humans are still providing a lage part of the intelligence: defining the domain, defining what good performance on it looks like, carefully designing model architectures, collecting and labeling large datasets, etc. This is fine for narrow task-specific problems, but is not really the end-all of AI, and does not even seem to work well on all well-defined tasks. As an example of another kind of inference, how about mathematical reasoning? I purposely pick one here that is seemingly very formal; should be possible for a computer to do it. Mathematicians are somehow able to invent conjectures and prove theorems without first being exposed to terabytes of labeled mathematical facts. The scientific method is kind of an even messier version of this. Or to take something laypeople do, people can usually learn games to at least a passable level from just a handful of playthroughs, not AlphaGo-style millions of plays (imagine if you had to play even 1,000 games of MT:G before you got the basic hang of it...).
All this kind of stuff is quite well-represented in the literature though, if you mean the scientific literature. Pop-press AI writing tends to cover a pretty specific subset of what's going on at AI conferences.
Yes. That's also my total focus right now. Happy to discuss, email on profile.
If, on the other hand, they are talking about making progress on the more traditional dream of AI, the focus on building these data sets seems to be a sad way to lock us into a local maximum for a long time.
Massive curated data sets are crutches that lead us into narrow-minded hyper-specialized systems. Are any of these funds investing in people working on systems that try to make sense of raw sensory data streams? I don't think we're ever going to move away from data sets by creating more and better data sets.