By itself it's unlikely to ever be knowledge of course.. i see it more akin to NLP than knowledge. Which is to say, a general purpose language parsing tool which we can hand the result to something else. A conversational API, if you will, but we'll still need layers to actually run logic. To know math if you will.
Disclaimer: I know very little on the subject. Pure speculation.
But, even if we drop that interesting edge case i suspect we can make something very useful with the primitive that LLMs offer.. in the calculator example. ChainLang and co seem a really interesting tool for LLMs.
(Assuming that this happens, of course. Diminishing returns could make scaling infeasible past some point, for instance.)
The LLM approach may not be able to replicate the "knowledge" your calculator has, but it (or some pre/postprocessor) may be able to recognize that a given question is actually something a calculator can answer concretely, and then it can delegate the computation to traditional software that really does "know" how to answer the question.
>Our brains have different processing centers.
Uhh, no they don't? Did you know everything you know now about math when you were born, and are you also incapable of learning new things about math? Because that's how the wolfram plug-in works.
The far more practical (profitable) outcome for what they currently built is to just make a useful tool, a "smarter" wolfram alpha, and that can be iterated upon by delegating relevant operations to specific techniques that are more applicable to the question at hand.
The map is still not the territory, sorry.
From https://en.wikipedia.org/wiki/Chinese_room#Complete_argument the conclusion of the complete argument is:
> (C1) Programs are neither constitutive of nor sufficient for minds.
> This should follow without controversy from the first three: Programs don't have semantics. Programs have only syntax, and syntax is insufficient for semantics. Every mind has semantics. Therefore no programs are minds.
---
I personally don't agree with it and believe that there is a flaw in:
> (A2) "Minds have mental contents (semantics)."
> Unlike the symbols used by a program, our thoughts have meaning: they represent things and we know what it is they represent.
While a person may know what they are thinking, examining the mind from the outside it isn't possible to know what the mind is thinking. I would contend that from the outside of a mind looking at the firings of neurons in a brain it is equally indecipherable to the connections of a neural net.
The only claim that "we know what it is they represent" is done from the privileged position of inside the mind.
I would argue that intelligence is more related to the Kolmogorov complexity exhibited by something.
( David Dowe: Minimum Message Length, Solomonoff-Kolmogorov complexity, intelligence, deep learning... https://youtu.be/jY_FuQbEtVM?t=886 )
That the model of GPT is much smaller than its input.
The Chinese room lookup table is enormously large.
If we attempt to relegate GPT as no better than a Chinese room, we can show that the Chinese room look up table is impossible with the amount of data that GPT has access to as part of its model.
If we say that its not a lookup table but instead an enormously complex interplay of inputs and variables, then the distinction between the room that GPT exists in and our own mind breaks down trying to distinguish which is which.
If we want to switch to consciousness, then possibly the argument can progress from there because GPT doesn't have any state once it is run (ChatGPT maintains state by feeding its output back into itself and then summarizing it when it runs out of space). However, in doing this we've separated consciousness and intelligence which means that the Chinese room shouldn't be applied as an intelligence test but rather a consciousness test.
Are GPT 3 and 4 conscious? I'll certainly agree that's a "no". Will some future GPT be conscious and if so, how do we test for it? For that matter, how do we test for consciousness for another entity that we're conversing with (and its not just Homer with a drinking bird tapping 'suggested replies' in Teams ( https://support.microsoft.com/en-gb/office/use-suggested-rep... ))?
They implement a set of physical processes that, when operated and interpreted by humans, can be mapped into a subset of arithmetic. There's a correspondence.
Correspondence is the most useful way to think about it IMO. If there's a correspodence between what the machine does, and things we humans understand, then the machine, as a tool, is useful.
Understanding is a loaded word. It has implications beyond correspondence when humans use it; it has aspects of qualia, of fact vs fiction, of situatedness in a graph of comprehension, of consonance or dissonance with a set of other concepts, and so on.
LLMs in my opinion have a good "situatedness" for words and concepts, relative to other concepts. Qualia - consciousness - arguably doesn't matter. Fact vs fiction, they're very shaky on. Consonance vs dissonance, they're useless at - LLMs IME tend to flatter the prompt, constructing arguments in whatever direction a loaded question leads. There's little to no coherence there at all.
Unless you're using a definition of "understand" that implies conscience of self, I would argue that a calculator is a device that understands nothing except (a subset of) math. That's what makes a calculator reliable in ways that ChatGPT is not.
Philosophically speaking it could be argued that no software understands anything, but I think in the context of this discussion "understands" means "has a model of its context and the way one interacts with it", which is something a calculator (and plenty other software) definitely has and ChatGPT has not.