A stereotypical picture of an engineering approach without scientific knowledge would be a list of ways to do stuff combined with hints about how to vary the approach per-situation. It requires lots memorizing, trial-and-error and experts that often can't fully explain their reasoning. It's easy to believe bridge-building before Newton was like this though I'm not an expert. Present day AI sounds a lot like from what I've read (though I'm not an expert here either).
Edit: And yes, one could argue that the progress Newton ushered in merely replaced one list of models with a higher, more general list of models - yes, but that is how progress gone so far.
And, of course, if we had some really good mathematical framework for describing and reasoning about object recognition, we probably wouldn't need to turn to ML to solve it ;)
Feature extraction also works much better when you toss a lot of data and processing power behind it. So, a lot of progress is simply more data and computing power vs better approaches. Consider how poorly deep leaning works when using a single 286.
But that's true of people too. How quickly can you read upside-down?
If you trained on a mixture of upside-down and right way up images, and tested on upside-down images, performance wouldn't take that much of a hit.
Sure, the problem is we are more willing to ignore failures that are similar to how we fail. IMO, when we compare AI approach X vs. Y we need to consider absolute performance not just performance similar to human performance.
Deep learning for example gains a lot from texture detection in images. But, that also makes it really easy to fool.
If I had to describe the root cause of this problem it would be that humans process "problems" rather than "things" and we "learn" by building an ever growing mental library of problem solving algorithms. As we continue to "learn", we refine our problem solving algorithms to be more general than specific. Compare that to a deep learning AI that learns by building an ever greater data library of things while refining algorithms to suit ever more specific use cases.
I tend to imagine our brains works similarly. It's not that you have a single "network" in your brain that recognizes test from all angle, but your brain is a "general purpose" machine with many networks that work together. I think current deep learning techniques are great for discrete tasks, and the improvement needed is to have many networks that work together properly with some form of intuition as to what should be done with the information at hand.
As much as architecture research gets denigrated these days, MLPs aren't what set off the revolution.
How 'bout "creating models that can work with more dimensions of the problem domain than are conveyed by standard data labeling"?
I mean, we don't simply want AI but actually "need" it in the sense that problems like biological system are too complex to understand without artificial enhancements to our comprehension processes - thus to "understand the problem domain better" we need AI. If it's true that "to build AI, we need to understand the problem domain better", it leaves us stuck in a chicken-and-problem. That might be the case but if we're going find a way out, we are going to need to build tools in the fashion humans used to solve problems many times before.
As an aside, I think it's important that we find a way to examine and inspect how an ML model "works". If you have some neural network that does really well at the problem, it would be nice if you could somehow peer into it and explain, in human terms, what insight the model has made into the problem. That might not be feasible with neural networks, as they're really just a bunch of weights in a matrix, but this is practical for something like decision trees. Just food for thought.
An explanation/tutorial, with clean images of the process: https://github.com/tensorflow/tensorflow/blob/r0.10/tensorfl...
Google’s investigation of it’s GoogLeNet architecture: http://storage.googleapis.com/deepdream/visualz/tensorflow_i...
Now, I say somewhat because results can be visually confusing, ex Google’s analysis. Even then, we can see the progression of layer complexity as we go deeper into ImageNet. Plus, we can see mixed4b_5x5_bottleneck_pre_relu has kernels that seem to correspond with noses and eyes. mixed_4d_5x5_pre_relu has a kernel that seems to correspond with cat faces.
Mmmaybe,
It's tricky to articulate what pattern the data-scientist could see ... that an automated system couldn't see. Or otherwise, perhaps the whole "loop" could be automated. Or possibly the original neural already finds all the patterns available and what's left can't be interpreted.
What is interesting to note, now that above idea is considered, is that this process model itself belongs to the set of human-machine coordinations. Another process model is where low level human mind is used to perform recognition tasks too hard (or too slow) for machine to perform, for example using porn surfers to perform computation tasks via e.g. captcha like puzzles.
Long term social ramifications of all this is also interesting to consider as it motivates machines to breed distinct types of humans ;)
However, I would mention that there's a larger "overhead" than many realize to methods which work without the creator or the user understanding why. You have "racist" AI which don't undertstand that correlation may not be causation in questions like whether someone should be paroled or get a loan, you have the AIs subject to adversial attacks of various sorts, where not knowing why the AI works is also problematic, you have a situation where the target to match varies over time and so-forth.
Which adds up to AI having more dimensions to it than simply "working well" and "working less well". Indeed, AI is effectively ad-hoc statistics with result derived heuristically.
So in the process of "getting things right" exploring all sorts of things certainly sounds good, it seems like there's an "understanding gap" that needs to be closed and some broader model of what's happening would be useful but naturally there's no guarantee we can find one.
You just described all of software engineering.
bridges aren't catenaries and Euler and Lagrange gave us the Euler-Lagrange equations, not Newton.
Bridge building is mostly well-understood engineering. When you study Static Mechanics [0] you learn all sorts of Physics equations, including Newtonian Mechanics, that completely describe the forces and motions of a structure based on measurable physical properties of the materials used and details about the shapes of those materials.
When you get into Fluid Dynamics, things are different. You start to encounter a bunch of things like Reynolds number [1], which is a dimensionless value related to turbulence that you just have to look up for the particular fluids and velocities you're working with. This number is pretty well defined, but there are a lot of others and their definitions and meanings aren't nearly as clear as F=ma. Back when I was in school, particle simulations for turbulent fluids was just beginning to be feasible, so to design something you plugged in dimensionless constants and didn't worry about the unpredictable fine-details. An example of this is the wind blowing through a bridge's structure, and water flowing around its base. The equations don't give you exact forces that the turbulent air and water will exert; they give you more of an average over time. A simulation, if you can do it, can show you things (like resonance) that the equations won't show you.
Then there was Strength of Materials. Here, the big thing was the Factor of Safety [2]. This is solidly in the rule-of-thumb engineering camp. This is where the engineer says "I think two 16" steel beams would be sufficient... so lets use three 20" beams just to be sure." This is still the way a lot of engineering design is done, because the real world is never precisely known, and the factor of safety will save you when something unexpected happens.
[0] https://en.wikipedia.org/wiki/Statics
The "rule of thumb" engineering that you speak of made me remember the different constants that were taught we should just accept as is because, well, it is considered constant. Nevermind where the guy in the book got it from, this is what works and this is what people in the industry has accepted to be standard.
https://en.wikipedia.org/wiki/List_of_structural_failures_an...
Yus - bridges.
It started as a cluster of 16M CPUs having taught itself to recognize a cat 95% of the time after training on 1B google images.
And we are now at One-Shot Imitation Learning, "a general system that can turn any demonstrations into robust policies that can accomplish an overwhelming variety of tasks".
One Shot Imitation Learning
And it kinda does, but in an engineering way rather than a statistics way.
Like reinforcement learning from pixels is pretty new (i would be really interested if you have 10+year old citations), and pretty amazing. I've been looking at RL (through OpenAI gym) and realising that I "just" need to annotate a bunch of images and then train a network that will predict (fire/no fire in Doom from those pixels, and I can just add another network that builds some history onto this net (like an RNN) and this might actually work, is kinda amazing.
I'm still not sure I believe that it's always a good approach, but some of my initial experiments with my own (mostly image so far) data have been pretty promising.
The hype is pretty annoying though, especially if you've been interested in these things for years.
The bar to entry for these kinds of applications has been significantly lowered, which means we'll see more of it. I guess, in some sense, it's similar to the explosion of computer programs following the advent of personal computers (maybe, I haven't thought deeply about this part).
Using high speed hardware can allow someone to do 10's or scores of runs a day. If you are doing one every 2 weeks or so then it's really, really hard to make any progress at all because you daren't take risks. So the productivity of 80 a day vs 2 per month isn't just 100x it's lots and lots more.
Also as you say it's lowered the bar which means that teams can onboard grad students and interns and get them to do something that's useful - it may be trivial - but it's useful.
Print "It's a cat!"Uh, we need TWO (2), that's TWO numbers:
conditional probability of recognizing a cat when there is one (detection rate)
conditional probability of claiming there is a cat when there isn't one.
The second is the false alarm rate or the conditional probability of a false alarm or the conditional probability of Type I error or the significance level of the test or the p-value, the most heavily used quantity in all of statistics.
One minus the detection rate is the conditional probability of Type II error.
Typically we can adjust the false alarm rate, and, if we are willing to accept a higher false alarm rate, then we can get a higher detection rate.
With my little program, the false alarm rate is also 100%. So, as a detector, my little program is worthless. But the program does have a 100% detection rate, and that's 5% better than the OP claimed.
If focus ONLY on detection rate, that is, recognizing a cat when there is one, then it's easy to get a 100% detection rate with just a trivial test -- just say everything is a cat as I did.
What's tricky is to have the detection rate high and the false alarm rate low. The best way to do that is in the classic Neyman-Pearson lemma. A good proof is possible using the Hahn decomposition from the Radon-Nikodym theorem in measure theory with the famous proof by von Neumann in W. Rudin, Real and Complex Analysis.
My little program was correct and not a joke.
Again, to evaluate a detector, need TWO, that's two, or 1 + 1 = 2 numbers.
What about a detector that is overall 95% correct? That's easy, too: Just show my detector cats 95% of the time.
If we are to be good at computer science, data science, ML/AI, and dip our toes into a huge ocean of beautifully done applied math, then we need to understand Type I and Type II errors. Sorry 'bout that.
Are we learning yet?
Say, you have a kitty cat and your vet does a blood count, say, whatever that is, and gets a number.
Now you want to know if your cat is sick or healthy.
Okay. From a lot of data on what appear to be healthy cats, we know what the probability distribution is for the blood count number.
So, we make a hypothesis that our cat is healthy. So, with this hypothesis, presto, bingo, we know the distribution of the number we got. We call this the null hypothesis because we are assuming that the situation is null, that is, nothing wrong, that is, that our cat is healthy.
Now, suppose our number falls way out in a tail of that distribution.
So, we say, either (A) our cat is healthy and we have observed something rare or (B) the rare is too rare for us to believe, and we reject the null hypothesis and conclude that our cat is sick.
Historically that worked great for testing a roulette wheel that was crooked.
So, as many before you, if you think about that little procedure too long, then you start to have questions! A lot of good math people don't believe statistical hypothesis testing; typically if it is their father, mother, wife, cat, son, or daughter, they DO start to believe!
Issues:
(1) Which tail of the distribution, the left or the right? Maybe in some context with some more information, we will know. E.g., for blood pressure for the elderly, we consider the upper tail, that is, blood pressure too high. For a sick patient, maybe we consider blood pressure too low unless they are sick from, say, cocaine in which case we may consider too high. So, which tail is not in the little two set dance I gave. Hmm, purists may be offended, often the case in statistics looked at too carefully! But, again, if it's your dear, total angel of a perfect daughter, then ...!
(2) If we have data on healthy kitty cats, what about also sick ones? Could we use that data? Yes, and we should. But in some real situations all we have a shot at getting is the data on the healthy -- e.g., maybe we have oceans of data on the healthy case (e.g., a high end server farm) but darned little data on the sick cases, e.g., the next really obscure virus attack.
(3) Why the tails at all? Why not just any area of low probability? Hmm .... Partly because we worship at the alter of central tendency?
Another reason is a bit heuristic: By going for the tails, for any selected false alarm rate, we maximize the area of our detection rate.
Okay, then we could generalize that to multidimensional data, e.g., as might get from several variables from a kitty cat, dear, angel perfect daughter, or a big server farm. That is, the distribution of the data in the healthy case looks like the Catskill Mountains. Then we pour in water to create lakes (assume they all seek the same level). The false alarm rate is the probability of the ground area under the lakes. A detection is a point in a lake. For a lower false alarm rate, we drain out some of the water. We maximize the geographical area for the false alarm rate we are willing to tolerate.
Well, I cheated -- that same nutshell also covers some of semester 102.
For more, the big name is E. Lehmann, long at Berkeley.
Go for it!
Thus they work on "safe" projects.
I have met a lot of academics like this over the years, but I think your broader point might be that this is not possible today, which I agree with, and which is why I left academia (modulo personal situations).
In AI, we're not building rockets yet, but we have some really awesome and really powerful bonfires or whatever.
At least that's how I understood his point.
(And in anesthesia, when we do understand those things, we may very well look on our use today as barbaric or dangerous.)
If this could work with computer-trained models, that would be incredible too. What could a great speech understanding system teach us about language? What tricks from a facial-expression classifier could help autistic kids understand their friends?
The biggest deficiency in AI is that we still don't have artificial systems which simulate human thought with any fidelity. Sooner or later that's bound to become a focus of attention.
Kind of like having kids.
Anyone who doesn't know what I'm talking about should read 'Goedel, Escher, Bach', or 'Fluid Analogies'. I haven't read them in a long while, but I'm sure they're going to be relevant for decades, because they deal with the fundamental challenge of what it means to think. Backpropagation may be part of the puzzle, but the brain (and intelligence) is so much more than that.
http://www.popularmechanics.com/science/a3278/why-watson-and...
1) learn how the brain works 2) build a simulator
most current AI research skips step 1
I disagree that step #1 is important.
Consider the "Air-foil", which led to flight. In one sense, its an approximation of the wings of birds and other animals.
But ultimately, the discovery that the "Air-foil" shape turns sideways blowing wind into an upward force now called "lift" is completely different from how most people understand bird wings.
Bird Wings flap, but Airplane Air Foils do not.
--------
Another example: Neural Networks are one of the best mathematical simulations of the human brain (as we understand it, as well as a few simplifications to make Artificial Neural Networks possible to run on modern GPUs / CPUs).
However, the big advances in "Game AI" the past few years are:
1. Monte Carlo Tree Search -- AlphaGo (although some of it is Neural Network training, the MCTS is the core of the algorithm)
2. Counterfactual Regret Minimization -- The Poker AI that out-bluffed humans
There are other methodologies which have proven very successful, despite little to no biological roots. IIRC, Bayesian Inference is a widely deployed machine learning technique (for some definition of Machine Learning at least), but has almost nothing to do with how a human brain works.
An interesting field of AI is "Genetic Algorithms", which have biological roots but not anything based on the biology of brains, to achieve machine learning. Overall, a "Genetic Algorithm" is really just a randomized search in a multidimensional problem, but the idea of it was inspired by Darwinian Evolution.
AFAIK, this is not correct. Many of the Go playing algorithms before AlphaGo used MCTS or some variant. The true breakthrough of AlphaGo was deep reinforcement learning.
> AlphaGo's performance without search The AlphaGo team then tested the performance of the policy networks. At each move, they chose the actions that were predicted by the policy networks to give the highest likelihood of a win. Using this strategy, each move took only 3 ms to compute. They tested their best-performing policy network against Pachi, the strongest open-source Go program, and which relies on 100,000 simulations of MCTS at each turn. AlphaGo's policy network won 85% of the games against Pachi! I find this result truly remarkable. A fast feed-forward architecture (a convolutional network) was able to outperform a system that relies extensively on search. https://www.tastehit.com/blog/google-deepmind-alphago-how-it...
I don't know whether AlphaGo Master (the next version of AlphaGo that was trained purely with self-played games and has not been beaten in 60+ games) even uses MTCS.
That said, I agree that learning how the brain works seems unimportant and unnecessary. Evolution doesn't know how a brain works, but it's given us Einstein, Michelangelo, and conversations on HN.
It seems really important to learn how to build evolution into attempts at AI, given that evolution is the only known mechanism that leads to what we recognize as intelligence.
you use antropomorphy to reflect on your own standpoint. we don't know how the brain works? we can feel it and psychologist have a huge body of work concerned with the topic and that is already having influence on competition and fitness.
two out of three ain't bad
Not true; "lift" was well known for thousands of years, horizontal "lift" is how ships sail upwind. The breakthrough for the Wright bros was making something light enough to make use of this phenomenon vertically.
Besides, if you could simulate a human brain, you will end up with something that needs to sleep, something with limited and unreliable memory, something that gets bored and distracted, something emotionally needy, etc.
Then the extending of this chaotic, messy system is wildly unknown even if we could get a piece-for-piece replication to work. Such a thing would be of great benefit to medicine, but not really for AI to even start with until medicine is done reverse engineering it.
how do you plan to build artificial intelligence with no model of intelligence, without learning about important experiments in learning and memory , it's the complete ignorance that drives me crazy.
AI is not for specialists.
That said, I find when trying to solve a problem with ML techniques, it's better to use someone who knows the problem domain really well than someone who only knows ML really well. Someone who really understands the problem they're trying to solve can encode that knowledge into their models when training the system. While I've seen people who really know ML but lack the specific domain knowledge labor for weeks, coming back to me with "discoveries" that are already well known.
Take the best Roman engineer.
Translate a first year engineering paper on structures into Latin.
Ask Roman to sit said paper.
What will happen and why? The Roman chap will look very confused and will make statements (in Latin) about how stupid this stuff is and how it has nothing to do with proper engineering. The Roman will score 0. The why is that the understanding of structures and materials in the ancient world was artizanal, based on trade knowledge (often secret and hard to reproduce) and not systematic, based on the scientific method and inspectable or testable.
Currently we accept that knives, cabinets and sheds may be built or made using artisanal knowledge, we do not accept that apartment blocks, aircraft or automobiles are built this way. Society insists that these are built using systematic knowledge because otherwise they sometimes fall down or crash.
The systematic approach to aircraft is the best example - think how much civil air traffic there is now, and how rare air crashes are. The issues of subsonic flight have been systematically accounted for, right up to the point where we now see 1:2,000,000 crashes per flight.
Mechanical, aeronautical and civil engineering proceed in this way. Issues are discovered with mechanisms or structures or materials, these are characterized with scientific investigation, the characterizations lead to constraints and parameters that are required to be accounted for in future designs and old designs are re-evaluated in the light of the new knowledge.
Stating that you will build a new building in a certain way because domes are strong and concrete is strong would not cut the mustard in the modern world... The parthenon has stood for 2000 years, but how many similar structures collapsed after a few months?
David Duvenaud, an assistant professor in the same department as Hinton at the University of Toronto, says deep learning has been somewhat like engineering before physics. “Someone writes a paper and says, ‘I made this bridge and it stood up!’ Another guy has a paper: ‘I made this bridge and it fell down—but then I added pillars, and then it stayed up.’ Then pillars are a hot new thing. Someone comes up with arches, and it’s like, ‘Arches are great!’” With physics, he says, “you can actually understand what’s going to work and why.” Only recently, he says, have we begun to move into that phase of actual understanding with artificial intelligence.
Hinton himself says, “Most conferences consist of making minor variations … as opposed to thinking hard and saying, ‘What is it about what we’re doing now that’s really deficient? What does it have difficulty with? Let’s focus on that.’”