When I read about LLMs and massive amounts of training data fed to neural networks around 2020 I realized that time had finally caught up with him.
Having some built in knowledge of the world through pre-training is only part of the bootstrapping process.
So a medical doctor and a structural engineer can never be considered general intelligence separately?
What we have is spiky ASI. It's really good at a huge number of things that nobody can be good at simultaneously. It also sucks at some stuff that's really obvious and easy for us.
Why not? Cognitive capability usually refers to one's capacity to think, reason, learn, and process information. Presumably, on average, medical doctors and structural engineers are pretty comparable on that front. Cognitive capacity is not dependent on your posteriori knowledge.
What? All the time!
We've been seeing discussions on post-quantum encryption for a decade or two. The effects of nuclear fusion have been in the background of countless scifi stories, the idea of an extreme amount of abundant energy unlocks 100s of "that can't happen due to energy density" dead-end what-ifs.
Additionally, we're pondering the effects of AGI/SGI because a handful of extremely rich people have been telling us we'll all be out of all our jobs in x years, for almost 6 years now.
They've literally been sitting in press releases telling us "Enjoy that comfort because you won't be able to feed your family when you're outsourced. Your work and career will be worthless, and we're leading the way towards returning you to peasant life!"
Regardless of how out-of-reach AGI is right now, people will continue to work on it unless we end up in some Butlerian jihad situation. Just like how fusion is still out of reach, but far closer. They've got fusion reactors running for over a minute now! 10 years ago they could barely get the metaphorical engine to turn over.
Why wouldn't we ponder the effects? Do you walk through doors without consideration of what's on the other side?
I'm not convinced that increasing the scale will lead to AGI. A paradigm shift obviously could lead to AGI because it's unknowable what that might be and therefore least the predicable.
Recursive Self Improvement has often been cited as a possible venue, but no one currently knows how to do that with LLMs and Transformers.