Re:NLP’s generalization problem, and how researchers are tac...
by YeGoblynQueenne on Monday August 27, 2018 @04:34AM
(#17850193)
Yes, classification noise is an issue, but there are ways around it and
they're not particularly complicated. For instance, the simplest thing you can
do is repeated random subsampling, which is not a big deal given the high
sample efficiency and the low training times (seconds, rather than hours, let
alone days or weeks).
See for instance this work, where Metagol is trained on
noisy image data by random subsampling:
The DeepMind paper flags up ILP's issues with noisy data as a show stopper,
but like I say in my comment above, I disagree. The ILP community has found
various ways to deal with noise over the years since the '90s.
If you are wondering what the downsides are of Meta-Interpretive Learning, the
real PITA with Metagol for me is the need to hand-craft inductive bias. This
is not different to choosing and fine-tuning a neural net architecture, or choosing
Bayesian priors etc, and in fact might be simpler to do in Metagol (because
inductive bias is clearly and cleanly encoded in metarules) but it's still a
pain. A couple of us are working on this currently. It's probably impossible
to do any sort of learning without some kind of structural bias- but it may
be possible to figure the right kind of structure out automatically, in some
cases, under some assumptions etc etc.
I think there's certainly an "ideal system" that is some kind of "best of both
worlds" between ILP and deep learning, but I wouldn't put my money on some
single algorithm doing both things at once, like the δILP system in the
DeepMind paper. I'd put my money (and research time) on combining the two
approaches as separate module, perhaps a deep learning module for "low-level"
perceptual tasks and a MIL module for "high-level" reasoning. That's what each
system does best, and there's no reason to try to add screw-driving
functionality to a hammer, or vice-versa.
See for instance this work, where Metagol is trained on noisy image data by random subsampling:
https://www.doc.ic.ac.uk/~shm/Papers/logvismlj.pdf
The DeepMind paper flags up ILP's issues with noisy data as a show stopper, but like I say in my comment above, I disagree. The ILP community has found various ways to deal with noise over the years since the '90s.
If you are wondering what the downsides are of Meta-Interpretive Learning, the real PITA with Metagol for me is the need to hand-craft inductive bias. This is not different to choosing and fine-tuning a neural net architecture, or choosing Bayesian priors etc, and in fact might be simpler to do in Metagol (because inductive bias is clearly and cleanly encoded in metarules) but it's still a pain. A couple of us are working on this currently. It's probably impossible to do any sort of learning without some kind of structural bias- but it may be possible to figure the right kind of structure out automatically, in some cases, under some assumptions etc etc.
I think there's certainly an "ideal system" that is some kind of "best of both worlds" between ILP and deep learning, but I wouldn't put my money on some single algorithm doing both things at once, like the δILP system in the DeepMind paper. I'd put my money (and research time) on combining the two approaches as separate module, perhaps a deep learning module for "low-level" perceptual tasks and a MIL module for "high-level" reasoning. That's what each system does best, and there's no reason to try to add screw-driving functionality to a hammer, or vice-versa.