Re:Simple questions to help reduce AI hype
by graycat on Sunday July 10, 2016 @08:16AM
(#12065377)
Short reaction: No!
In a little more detail, the author is
talking about cases of "an amazing system"
but, really, has in mind a quite narrow
view of such systems and how they work and
can be, and have been, built. Really, the
author is looking at less than, say, 5% of
the great examples of how to build "an
amazing system".
In more detail:
The article has its definition of AI:
> In this article, I won't quibble over
definitions, simply taking the broadest
term as used in the media: artificial
intelligence is whenever a system appears
more intelligent than we expect it to be.
Okay, from my usual understanding of AI,
say, the peer-reviewed papers I published
in AI, this is a relatively broad
definition, but while considering this
article, I'll accept that definition.
But, oops, right away in the sentence
before we have
> If there's any promise I can make about
the field of AI,
So, the author is saying that AI is a
field. From the author's definition of
AI, that is a very broad field indeed!
Maybe the most broad of all?
Still I will try to consider the article:
So, my next concern is
> Ask what cases the system will fail on
> "Revealing the failure cases of your
models is a way of avoiding over-hype"
> Kevin Murphy at ICML 2015
Now what is AI is starting to look quite
narrow, narrow enough to have a "model"
and "failure cases".
So, oops: Consider a simple computer
program that plays the little matchstick
game Nim: There are two players. The
game starts with three piles of
matchsticks. For a move a player picks
a pile and removes from it as many
matchsticks as they want but at least one.
The players alternate moves. The player
who picks up the last matchstick loses.
Okay, there is a simple algorithms for
perfect play. Then if both players play
perfect games, who wins depends just on
who moves first. If only one player, A,
is playing a perfect game but on that
instance of the game with perfect play by
the other player, B, should lose, the
player A will still win as soon as player
B makes a mistake.
How to play a perfect game is in
Courant and Robbins, What is
Mathematics.
So back to the last quote, there are no
real "failure cases" and, really, no
"model". Instead, there's some applied
math.
But, I'll keep reading:
> No AI system yet developed will
magically fit every use case.
Well, the little algorithm for playing Nim
fits "every use case".
> If a researcher tells you that a model
got state of the art results out of the
box, they're either exceedingly lucky or
giving you a fairy tale version of
reality.
No. As in a program to play perfect Nim,
there is another explanation: They have a
perfect solution for the problem.
Again, by this far into the article, the
author seems to have in mind a limited
view of what AI is.
> Even if their model is arbitrarily
flexible, there are still fundamental
limits placed on us by information theory.
Not for a program written using the
Courant and Robbins algorithm.
Information theory has essentially nothing
to do with it.
Continuing:
> "How much training data is required?"
For the Nim program, none.
> "Can this work unsupervised (= without
labeling the examples)?"
Well, since there are no examples,
apparently so.
> "Can the system predict out of
vocabulary names?"
WHAT? I doubt I understand the question.
But with the Nim program, maybe so.
The author seems to be assuming that the
program is about natural language
understanding. Not all computer programs
are.
> "How stable is the model's performance
over time?"
With the Nim program, perfectly stable.
> Very few datasets are entirely
representative of what the task looks like
in the real world.
"Dataset"? The Nim program doesn't have
one.
Again, it appears that the author is
thinking of a narrow view of AI, much
more narrow than the definition the author
gave.
> Any claim of advanced research without
publication is suspect at best
> "If you do research in isolation, the
quality goes down. Which is why military
research sucks."
> Yann LeCun at ICML 2015
Uh, sorry, but I've done all my original
research "in isolation" -- in a small,
quiet room, sometimes on my back looking
at the ceiling. For my Ph.D. dissertation
in applied math, the first of the research
was on an airplane ride from NY to TN.
The rest of the research was independently
in my first summer in graduate school.
> The rate of change in the field of AI is
such that anyone on the sidelines is at
best keeping up.
No, not really: I call my research
applied mathematics, but it meets the
author's definition of AI. But I pay
essentially no serious attention to the
field of AI (e.g., the work of people
who claim to be in AI) at all. And the
number of researchers in the field of AI,
computer science, and information
technology entrepreneurship that have the
mathematical prerequisites to read my
research is tiny. So, I'm not paying
attention to the field of AI, and AI
researchers are paying no attention to my
work, some of which is published and, in
one case, for one important problem,
really, a significantly large set of
important problems, totally knocks the
socks off anything in AI. That case?
Sure, some applied math, right, complete
with theorems and proofs. In more detail,
the work is some new results in
mathematical statistics.
> It is certainly not impossible for an
entity to come out of nowhere with an
amazing system but not it is far less
likely.
Naw! Essentially the opposite is true:
If want "an amazing system", usually stay
far away from AI -- just totally ignore
the field. Instead, pursue applied math.
E.g., molecular spectroscopy for
identifying chemical molecules is terrific
stuff, but is has essentially no "training
data". Instead, the work is based on some
quantum mechanics and group theory.
Again, the author is thinking in a way
quite narrow compared with the author's
definition of AI and compared with what is
done to build "an amazing system".
> It also means they haven't been put
through the standard evaluations that
academia would have placed on them.
Well, as I've seen in some of my
peer-reviewed publications, my work in
applied mathematics is more advanced and
with standards commonly higher, e.g.,
theorems and proofs, than the academic
review process is able to follow.
Besides, any work I publish means I've
given away the intellectual property, and
now mostly I don't want to do that.
> AI won't save a broken business plan.
An easy upper bound is asking if the
business plan would work with free human
labor replacing the automated component.
Going back decades, for many cases of "an
amazing system", the idea that human labor
could do the work at all is essentially
absurd. Here the author is being narrow
again.
> Achieving human level performance on any
real world task is an exceedingly
difficult endeavor when moving away from a
few simple and well defined tasks.
No: Totally knocking the socks off human
level performance is very common in
applications of applied mathematics to
complicated real problems, e.g., critical
mass calculations for nuclear weapons,
target detection via phased array passive
sonar, also with Wiener filtering, to
track the target, maybe with Kalman
filtering, differential game theory for
saying how a missile should catch a
fighter plane, deterministic optimal
control theory for, say, least time
control to get an airplane to a given
altitude, matching anti-ballistic missiles
to incoming warheads, a huge range of
applications of deterministic
optimization, e.g., linear programming and
integer linear programming, going way back
to, say, even vacuum tube computers.
> All of this is to say that if the
business plan doesn't work with free
humans, AI won't save it.
Well, for my startup, there's no way
humans could do the crucial, core applied
mathematics, but my software based in my
original applied math works just fine!
Again the author is thinking about cases
of "an amazing system" much more narrow
than the author's definition of AI and
much, much more narrow than good work in
applied math going back at least 50 years.
In a little more detail, the author is talking about cases of "an amazing system" but, really, has in mind a quite narrow view of such systems and how they work and can be, and have been, built. Really, the author is looking at less than, say, 5% of the great examples of how to build "an amazing system".
In more detail:
The article has its definition of AI:
> In this article, I won't quibble over definitions, simply taking the broadest term as used in the media: artificial intelligence is whenever a system appears more intelligent than we expect it to be.
Okay, from my usual understanding of AI, say, the peer-reviewed papers I published in AI, this is a relatively broad definition, but while considering this article, I'll accept that definition.
But, oops, right away in the sentence before we have
> If there's any promise I can make about the field of AI,
So, the author is saying that AI is a field. From the author's definition of AI, that is a very broad field indeed! Maybe the most broad of all?
Still I will try to consider the article:
So, my next concern is
> Ask what cases the system will fail on
> "Revealing the failure cases of your models is a way of avoiding over-hype"
> Kevin Murphy at ICML 2015
Now what is AI is starting to look quite narrow, narrow enough to have a "model" and "failure cases".
So, oops: Consider a simple computer program that plays the little matchstick game Nim: There are two players. The game starts with three piles of matchsticks. For a move a player picks a pile and removes from it as many matchsticks as they want but at least one. The players alternate moves. The player who picks up the last matchstick loses.
Okay, there is a simple algorithms for perfect play. Then if both players play perfect games, who wins depends just on who moves first. If only one player, A, is playing a perfect game but on that instance of the game with perfect play by the other player, B, should lose, the player A will still win as soon as player B makes a mistake.
How to play a perfect game is in
Courant and Robbins, What is Mathematics.
So back to the last quote, there are no real "failure cases" and, really, no "model". Instead, there's some applied math.
But, I'll keep reading:
> No AI system yet developed will magically fit every use case.
Well, the little algorithm for playing Nim fits "every use case".
> If a researcher tells you that a model got state of the art results out of the box, they're either exceedingly lucky or giving you a fairy tale version of reality.
No. As in a program to play perfect Nim, there is another explanation: They have a perfect solution for the problem.
Again, by this far into the article, the author seems to have in mind a limited view of what AI is.
> Even if their model is arbitrarily flexible, there are still fundamental limits placed on us by information theory.
Not for a program written using the Courant and Robbins algorithm. Information theory has essentially nothing to do with it.
Continuing:
> "How much training data is required?"
For the Nim program, none.
> "Can this work unsupervised (= without labeling the examples)?"
Well, since there are no examples, apparently so.
> "Can the system predict out of vocabulary names?"
WHAT? I doubt I understand the question. But with the Nim program, maybe so.
The author seems to be assuming that the program is about natural language understanding. Not all computer programs are.
> "How stable is the model's performance over time?"
With the Nim program, perfectly stable.
> Very few datasets are entirely representative of what the task looks like in the real world.
"Dataset"? The Nim program doesn't have one.
Again, it appears that the author is thinking of a narrow view of AI, much more narrow than the definition the author gave.
> Any claim of advanced research without publication is suspect at best
> "If you do research in isolation, the quality goes down. Which is why military research sucks."
> Yann LeCun at ICML 2015
Uh, sorry, but I've done all my original research "in isolation" -- in a small, quiet room, sometimes on my back looking at the ceiling. For my Ph.D. dissertation in applied math, the first of the research was on an airplane ride from NY to TN. The rest of the research was independently in my first summer in graduate school.
> The rate of change in the field of AI is such that anyone on the sidelines is at best keeping up.
No, not really: I call my research applied mathematics, but it meets the author's definition of AI. But I pay essentially no serious attention to the field of AI (e.g., the work of people who claim to be in AI) at all. And the number of researchers in the field of AI, computer science, and information technology entrepreneurship that have the mathematical prerequisites to read my research is tiny. So, I'm not paying attention to the field of AI, and AI researchers are paying no attention to my work, some of which is published and, in one case, for one important problem, really, a significantly large set of important problems, totally knocks the socks off anything in AI. That case? Sure, some applied math, right, complete with theorems and proofs. In more detail, the work is some new results in mathematical statistics.
> It is certainly not impossible for an entity to come out of nowhere with an amazing system but not it is far less likely.
Naw! Essentially the opposite is true: If want "an amazing system", usually stay far away from AI -- just totally ignore the field. Instead, pursue applied math. E.g., molecular spectroscopy for identifying chemical molecules is terrific stuff, but is has essentially no "training data". Instead, the work is based on some quantum mechanics and group theory.
Again, the author is thinking in a way quite narrow compared with the author's definition of AI and compared with what is done to build "an amazing system".
> It also means they haven't been put through the standard evaluations that academia would have placed on them.
Well, as I've seen in some of my peer-reviewed publications, my work in applied mathematics is more advanced and with standards commonly higher, e.g., theorems and proofs, than the academic review process is able to follow.
Besides, any work I publish means I've given away the intellectual property, and now mostly I don't want to do that.
> AI won't save a broken business plan. An easy upper bound is asking if the business plan would work with free human labor replacing the automated component.
Going back decades, for many cases of "an amazing system", the idea that human labor could do the work at all is essentially absurd. Here the author is being narrow again.
> Achieving human level performance on any real world task is an exceedingly difficult endeavor when moving away from a few simple and well defined tasks.
No: Totally knocking the socks off human level performance is very common in applications of applied mathematics to complicated real problems, e.g., critical mass calculations for nuclear weapons, target detection via phased array passive sonar, also with Wiener filtering, to track the target, maybe with Kalman filtering, differential game theory for saying how a missile should catch a fighter plane, deterministic optimal control theory for, say, least time control to get an airplane to a given altitude, matching anti-ballistic missiles to incoming warheads, a huge range of applications of deterministic optimization, e.g., linear programming and integer linear programming, going way back to, say, even vacuum tube computers.
> All of this is to say that if the business plan doesn't work with free humans, AI won't save it.
Well, for my startup, there's no way humans could do the crucial, core applied mathematics, but my software based in my original applied math works just fine!
Again the author is thinking about cases of "an amazing system" much more narrow than the author's definition of AI and much, much more narrow than good work in applied math going back at least 50 years.