To your first point I think that's actually where we are. Poking in the dark with no real understanding of the tools at our disposal. And the algorithms DO have power. We don't know WHY, except that we are emulating solutions given to us by biological evolution.
To your second point, that's basically what evolution did. Poked in the dark for several billions of years and ended up with solution machines (us).
Re:AGI Has Been Delayed
by FeepingCreature on Thursday December 05, 2019 @03:36AM
(#21710820)
Yes, I'm aware, that's in large part why it worries me. And I think you undersell the importance of online learning. NNs are far from "plug in and forget."
The aphorism used to be "hard things are easy and easy things are hard". Ie. arithmetic with large numbers is trivial, but opening a door or recognizing a label is the most difficult problem in the world. We're starting to close in on reliable, possibly prepackaged solutions for the latter; on the other hand, understanding the logic of intelligence turns out to have some teeth after all. Reality would do us a great favor if building genuine intelligence required actual understanding and knowledge of how to build reliable cognition; unfortunately our precedent suggests it's mostly done by kludging a half-baked barely-debugged motivation system onto a big heap of instinct-driven neural networks.
I'm worried some spark will get the bright idea to combine some approximating variant of AIXI and a neural network for theory selection...