Gödel's incompleteness theorems
https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...
Gödels theorems are about formal axiomatic theories. To apply them to human intelligence, you'd have to prove that human intelligence springs from formal axiomatic theories. I don't think this is possible, which would mean that you can't apply the theorems.
> No matter how much data we give it, no matter how big it is, it’ll never be “human intelligent” since it’s impossible for us to describe a loss function for being human or describe being human in a dataset.
How do you know? If we were able to fully record whatever is going on in someones brain, we should be able to build a loss function for it. How do you know that this is fundamentally impossible?
> We’ll never be able to evaluate it, since we can’t fully describe what it means to communicate because to do that we’d need to communicate it and that process can’t be fully self describing.
Why not? Again, if you argue that this is due to Gödels theorems, you'd have to prove that our communication itself is based on formal axiomatic theories.
It applies to everything we could say or think or measure about intelligence (and anything else). All that probably "spring from" things that don't (rely on our axioms), but they are not accessible to us, so that doesn't really help.
How do you know? Gödels proof doesn't support your claim since it only applies to formal axiomatic theories. Do you have an alternative proof for his theorems also applying to all other systems?
Maybe gravity makes things fall down, sure, but maybe there are tiny kobolds in the spaces between all particles with little clipboards that calculate the correct motion and cast spells to move them. I'm not trying to be a smartass, but I honestly tried and could not find bedrock. Can you name (or even just think) something that doesn't rest on something else or an assumption? I honestly can't.
> Gödels proof doesn't support your claim since it only applies to formal axiomatic theories.
"only"? I'd say those axioms are a superset of the sloppy stuff we throw around in our day to day, like "this is a chair"; if we drilled down on our informal speech and thoughts, we'd at best arrive at such axioms, which ultimately rest on things we simply posited (because otherwise there would be nothing to think about, and no way to think about it -- I'm not knocking it per se, just the idea that the quest for truth could possibly ever be complete, which makes it no less noble IMO).
I'm not sure how a "bedrock" relates to the question whether an intelligence can ever fully describe what an intelligence is. When answering this question, we don't need to find a "natural" bedrock, since the assumptions we choose are the bedrock we build on. As long as those assumptions align with reality to the best of our knowledge and the end result passes all tests we can think of, what does it matter whether there might be more to know? Of course it doesn't mean we should stop searching, but it also doesn't mean we should not even try. There are many such unfalsifiable statements, but that doesn't mean they stop us from answering other questions.
> "only"? I'd say those axioms are a superset of the sloppy stuff we throw around in our day to day, like "this is a chair"; if we drilled down on our informal speech and thoughts, we'd at best arrive at such axioms, which ultimately rest on things we simply posited (because otherwise there would be nothing to think about, and no way to think about it -- I'm not knocking it per se, just the idea that the quest for truth could possibly ever be complete, which makes it no less noble IMO).
I don't think this is true, and if you can prove it, you might earn a Nobel prize. "Formal axiomatic theories" are well-defined - as Wikipedia states, they are "formal systems that are of sufficient complexity to express the basic arithmetic of the natural numbers and which are consistent and effectively axiomatized. [...] In general, a formal system is a deductive apparatus that consists of a particular set of axioms along with rules of symbolic manipulation (or rules of inference) that allow for the derivation of new theorems from the axioms."
Can you try to describe how you'd "drill down" on informal speech to transform it into such a system? There are many, many examples for systems that are absolutely not based on formal axiomatic systems.
As I said, just keep asking "why?" or "what does that mean?", then repeat that with the answer. Sooner or later you hit an assumption and a shrug. I wouldn't understand Gödel's proof even if I tried to, I'm sure -- it "rings true" because it matches my own intellectual observations regardless where I turn.
> As long as those assumptions align with reality to the best of our knowledge and the end result passes all tests we can think of, what does it matter whether there might be more to know?
It matters for the question whether you can fully describe a system from within that system, that's all. But I'd argue even whether we made an effort or no effort, whether it passes all the tests we came up with or doesn't, doesn't really matter (in regards to that question) because any ground we cover won't bridge what remains an infinite distance. I still think it's good, but it's more like going for a walk each day: you always arrive where you started out, you're just getting fresh air and what other temporary benefits come with it. It beats just staying where you started out.
I can't find the quote but apparently Werner Heisenberg said something along those lines, that we basically set out to find the bed rock of reality, but more and more are just facing ourselves, that is, our instruments of measurement and ways to conceptualize things. And again, I don't know jack about quantum mechanics and don't want to call on the authority of Heisenberg and Gödel. But I hear they know their fields, right, and it matches everything I know in any area, both the ones I am bad in and the ones I am really bad in.
I'm not saying it's a problem, just that that's how it is. But thinking you know the ultimate and final truth because it passes all tests (e.g. witches sink), and thinking software is actually intelligent because it convinces you it is, when it really isn't, can be super mega dangerous. And comments how picking the statistically most likely word is "basically what our brain does" [0] etc. show an even worse possibility; where we take the shortcut of just confusing what we are creating with us, because then it's easy and now we know how we think (when we really don't, not remotely). It just generally seems backwards to start out with the goal of "AI" when we can't even describe what we're looking for, much less how to build or find it. Having no more than "we'll know it when we have it", plus eagerness to claim we have it, is a recipe for at least a lot of circus, if not disaster.
[0] And that's on HN. Now ponder, for example, the average opinion of HN on say, whether banks should limit your passwords in all sorts of weird way that imply they're not hashing them, and how much worse the "real world" is. In this case, even the people at the forefront are so keen to move fast and break things, so the "real world" is pretty much doomed I'd say.
> As I said, just keep asking "why?" or "what does that mean?", then repeat that with the answer. Sooner or later you hit an assumption and a shrug.
I still don't understand your assumption. Do you think that any axiomatic system is a formal axiomatic theory? As I've said before, this term is well-defined, and I don't see how you could "drill down" on natural language to arrive at such a system. There are many axiomatic systems that are not formal axiomatic theories, and Gödels proof doesn't apply to those.
> I wouldn't understand Gödel's proof even if I tried to, I'm sure -- it "rings true" because it matches my own intellectual observations regardless where I turn.
This is why I've been asking about how you'd bridge the gap between Gödels proof and your assumption, because Gödels proof applies strictly to one thing, and you seem to apply it to everything, even if it doesn't meet the requirements of the proof. But I guess you're arguing from a philosophical standpoint, not a logical one.
If it was a nail in the coffin for AGI, then humans, who have exactly the same limit, wouldn't count as a general intelligence.
While such a definition would be possible, I don't think it's useful.
It sounds like maybe you're arguing that humans will never be able to conclusively determine if an artificial intelligence is equivalent to a human intelligence, on the basis of a theory that a human can't describe precisely what it is to have human-equivalent intelligence from inside the system of a human brain. Humans build things that are too complex for any one person to hold the entirety of in their conscious mind all the time, by working together, or by organizing it into simpler pieces. But even setting that tangent aside, if your theory is accurate, would you accept that an AGI could prove it was more intelligent than a human by successfully describing human intelligence and how to create an equivalent AGI?
Because we’re unable to make a perfect, totally correct metric, I find it unlikely that any of the current generation of AI will get anywhere near human level.
Again, not that these new models aren’t extremely useful or impressive, but not really “intelligent” as a human is.
Technically, this is also a nail in the coffin for humans. Humans are also inside a system.
Even back with Kant, humans have been trying to figure out how to see beyond their own systems.
That’s like saying a physics simulation is basically an entire sub universe on your computer.
It sounds true, but it’s just not. It’s a gross oversimplification
I think Gödel had a proof for how it’s impossible to fully describe a system from within that system. That’s the nail in the coffin for AGI.
No matter how much data we give it, no matter how big it is, it’ll never be “human intelligent” since it’s impossible for us to describe a loss function for being human or describe being human in a dataset.
We’ll never be able to evaluate it, since we can’t fully describe what it means to communicate because to do that we’d need to communicate it and that process can’t be fully self describing.
Not to say AI isn’t useful or impressive, but it’ll never be comparable to humans, truly.