I don't think I have any basis for this besides a gut feeling and some daydreams, but I think each of the major methodologies like reinforcement learning, transformers, graph nns etc need to be combined into some larger type of ensemble and worked together into a cohesive system with feedback loops for online and offline learning for a shot at AGI.
I've been doing ML projects for like 6 years, mostly in NLP but have dipped into reinforcement learning because it interests me and my gut feeling has been much that there are a lot of complimentary learning systems that can handle different problems really well and cover for limitations in others and I'd like to see what happens if we smash them together towards the goal of generally beating baselines for as many benchmarks as possible.
Or I just hope there is something there. Unique. Cannot be copied. It might not last or re-birth or incarnations… n fact it might be better that way.
Ok, continue the rationalist approach, great find no doubt. And in fact nn may not even a total rational but a data/probabilist model. But just hope it is incomplete.
Shower thought that just came up: observe that while we're endlessly chasing a bigger-than-reality String Theory, actually working physics implemented in real-world machines follow the specialized approach of one partial but perfect theory for each category of problems. We build hybrid, because as far as we can tell, reality is variations, and the biological probably most of all. So in trying to build a being…
- in real life, graphs are not fixed
- you need to deal with the many different potential representations of the graph (permutation invariance)
- the nodes are usually containing more features than a single scalar value
but this is definitely not the best explanation, I think this guy does a lot better job: https://youtu.be/JtDgmmQ60x8
Since writing A Thousand Brains, Jeff Hawkins has revealed fascinating structures within the brain, a finite set of structure 'types' so to speak (families of similarly architectured brain parts).
Graphs are definitely part of the biological design, but in taking inspiration from nature to build our own beings, we should take notice that the real thing is vastly more complex, and investigate more exhaustively the ins and outs of real brain structures.