So we don't know, it just sounds plausible.
However, I do think there is an important difference between "in principle" and "in practice".
I agree that there does not seem to be any fundamental obstacle that would necessarily prevent it. But at the same time, I'm not convinced it will ever be achieved.
I at least can't come up with any convincing specific reasons. That doesn't mean it doesn't exist of course, but given the amount of progress we've made on tasks we previously believed to be equivalent to general intelligence, it's difficult for me to see any clear reason why it shouldn't be possible sometime in the next century or two.
We have a rough idea of the quantity of computation the brain is doing, so either the brain is using computational methods beyond Turing completeness, or the only piece we're missing is the algorithm (since we're within reach of the raw FLOPs). If it's "just" an algorithm that we have to search for, it seems extremely difficult to predict when that may or may not come.
> In my opinion, we won't ever have generalized AI. Or at least, we might not ever agree that we have it.
But I disagree here. Eventually we’ll have something that can do everything we can do, just better, and at that point we might still be unable to draw a line, but we’ll at least be able to agree that a superset of human intelligence counts as AGI.
We don’t need to define consciousness or intelligence concretely to have things that most people would agree are intelligent.
My own personal definitions of intelligence cover some amount of self-reinforced learning. That is, if I tell an AI how to do something, it will think about how to do it, Google for the answer, watch and understand a YouTube tutorial, and try to follow along with it, and if it doesn't understand a part it has the ability to rewatch and try again.
I've never seen a jaguar in person; I've maybe seen 10 minutes total of videos and photos of one, and yet I can imagine one running through a field, chasing down a predator, and eating its corpse. Current AI need to be trained on gigabytes of data manually labeled "jaguar" to identify a single one. Why can't an AI, after seeing something it doesn't recognize, ask a question like "what is that"?
Just a few years ago, there was that Minecraft AI challenge that failed spectacularly -- the task was to watch 50 hours of video of someone else playing the game, and then do it themselves. I could probably do that after ten minutes.
They aren't human level yet in all tasks but we are getting demonstrably closer. I would spend more time studying what exists today and less time critiquing
DALL-E2 is a fantastic example of zero-shot learning working spectacularly well. It's able to combine various concepts interchangeably.
Who told you we cannot.
Oh, but there is no need for such """correctness""".
The way is also well defined negatively - the lacks express their presence clearly -, and this page provides already a few examples:
-- "A is taller than B, B plays basketball, who is taller?"
If the system does not reply "A", then it does not understand the question; by a general consideration, it may be evident that it was designed to "guess" instead of "know", and there lies very patently the structural fault.
-- "A just left the house and is now in the market, where is A?"
If the system does not understand "states", etc. If it cannot build a world representation, etc.
-- "A fell from the flying helicopter, how is A?"
If the system does not understand "falling from an height", etc. If it cannot "tell itself a story" (Prof. Patrick Winston), etc.
The latter tells you about the logical relations between those "definitions" (of conjunction, so that (¬Av¬B)→¬C ). Prof. Winston defines "intelligence" as "the ability to tell oneself stories, which will also reveal outcomes of experiences you have not concretely lived". The definition may leave gaps when taken alone (especially if by somebody who confuses indications and definitions), but the lack of what is intended with such idea has full impact (no """inner storytelling""" - actually, I would say, no "productive engine" -, no "intelligence").
(And honestly, while the base attitude of course remains to be "helpful", for current lack of a better term, sometimes - fortunately very occasionally - to provide a proper "complete" reply ceases to be economically viable.)
Of course, the issue is that the way we actually solve these problems is by coming up with a clever search or brute force strategy (chess, self-driving, poker), and yet we're no really closer to a generalized AI. Part of this is philosophical -- we can't define things like consciousness or intelligence in concrete terms, and so we have no real goals to work towards. We keep hoping that stumbling around in one of these other areas will lead us in the right direction, but so far, it just hasn't. The goalposts aren't moving, we simply haven't found where they are.
In my opinion, we won't ever have generalized AI. Or at least, we might not ever agree that we have it. If 3,000 years of philosophers bickering hasn't given us any true insights yet, it's probably not going to suddenly show up now. But for now, weeeeee!!! Enjoy the funding before the next AI winter kicks in! Look at all the cool new progress happening! But to paraphrase someone who's name I forgot, "If your goal is to drive from San Francisco to the Moon, making it to Boston sure looks like you've made a lot of progress."