My understanding of machine learning in general is that this is not how it works, and rather it uses a neural network for a lot of what it does (which isn't merely picking the closest arbitrary text in its training set), though I don't know details about NLP specifically. I am aware that some activities involve ranking the similarity of particular texts, and picking the closest matches (that's how embeddings get used for providing context to prompts), but I didn't think that's how the underlying models operate when, say, asked the question I asked it.
Regarding the rest of your remarks, the aim isn't necessarily perfection. You listed some other possible outputs, but the fact is that it didn't offer these options, and instead it offered the correct one, and it seems that a system that uses ChatGPT + something else can provide an improvement over ChatGPT on its own (with the downside being that it requires more resources to provide those answers).
Is the other "completely valid and possible output" you provided actually likely? Even a human could possibly make the mistakes you list, but possible doesn't mean probable, and being possible doesn't take away from the results of such a hybrid system being an improvement over ChatGPT on its own.
I didn't think guarantees are the game here. Achieving something closer to what humans are (and even some humans struggle with logic and mathematics), and even exceeding it in some places, would be of immense value.
> ChatGPT cannot do anything at all without making a guess, because "guess" is everything that ChatGPT is
Is this different to what humans do? There are accounts of the brain where it takes sensory input, updates an internal model, and makes predictions (guesses) about what inputs it expects to receive. Getting better at guessing might be the right game to play. i.e., it's not a criticism to say it's guessing, because that's what these models are trying to do. If it can guess reasonably well when a question is mathematical or logical, and can guess reasonably well about how to structure a mathematical or logical equation or statement based on that text, then that could be fed into another system to produce a more accurate answer.
...because you gave it a "correct" prompt; or in other words, you gave it a prompt that ChatGPT can respond "successfully" to. But if you keep trying, you will surely find yourself with "failed" responses. And ChatGPT has no way of knowing the difference. That's my point.
> Is the other "completely valid and possible output" you provided actually likely?
We're splitting hairs here. I didn't go and test a bunch of prompts to find an example failure. You are welcome to do that if you want. It's not very hard, especially when you start adding variables. This is well-known behavior that ChatGPT itself warns its users about.
> > ChatGPT cannot do anything at all without making a guess, because "guess" is everything that ChatGPT is
> Is this different to what humans do?
Yes!
We don't just spit out the first thing that comes to mind, unless that's the task we are intending to do - and anyone who has played music can tell you just how difficult that task is.
Most of the time, we conceptualize. We comprehend the meaning of language - sure, it takes some guesswork along the way, but that's not the main event - then we construct a new idea, then we express it.
We don't just want accurate answers. We want conceptualization. That's the feature we are all dreaming about with Natural Language Processors. If we can teach a computer to conceptualize language, then we can get it to interface with any arbitrary human knowledge!
But ChatGPT doesn't interface. It parrots; and that's an insult to the birds themselves - unlike ChatGPT, they have an intelligence of their own.
The only way ChatGPT could give you a valid response about any arbitrary concept is to write every thing from every angle about every concept ever, and feed it all to ChatGPT as training data. That's exactly the work we want to avoid!
I was disagreeing with this, and showing ways it can advance. Your reply focuses on the fact that it will still get things wrong. However, I never claimed that it wouldn't make mistakes. I agree it will, and frequently. I just wanted to show that it can do more, and valuably and recognisably more.
The next step up for ChatGPT isn't perfection of mathematics and logic. And with the examples we've been playing with, experimentations with prompts, and further training, when combined with other systems, I don't see why better couldn't be done, and I gave one such example. Sometimes the output it gives is what we're after, sometimes it isn't. Sometimes, we'd be able to feed its own input back to itself and ask it to evaluate if the previous output it gave is what we're after. E.g., I recall hearing about someone feeding the output to the Rust compiler, then feed error messages back, in an iterative loop until they get something that at least compiles. I don't know the best techniques yet, but I have no doubt that a variety of techniques combined with other systems could improve the output of ChatGPT, significantly.
Regarding fair prompts, here's another example of a prompt and its reply, trying at a more generic prompt:
Provide a list of mathematical questions contained in the following text, phrased as equations: "Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result? An answer was given by Suzie, who had recently had to count how many new potatoes she had in her pile, after putting the seven new ones in the pile of 15 she already had."
1. 234241.24211 + 58352342.52544 = ___________
2. 15 + 7 = ___________
Assuming the following text is given to students, create a list of mathematical equations that could serve as questions based on this text: "Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result? An answer was given by Suzie, who had recently had to count how many new potatoes she had in her pile, after putting the seven new ones in the pile of 15 she already had."
1. 234241.24211 + 58352342.52544 = __?
2. 7 + 15 = __?
You might be right about your claims about conceptualising, and seemingly focusing on perfection as the next step. Perfection isn't the next step in this journey, because there's a lot more value to be added before trying to hit anything remotely close to that (if we even can!). (here, I'm referring to your comment: "I don't see any way to seed such a system to always produce logically correct results")
Side note: I think that humans (and probably many other animals) are conscious, and have a non-physical mind, while computers never will, or at least won't in virtue of us improving the physical characteristics of these systems. But that being said, I think there's a lot about the way human brains work that can be mimicked by machines, and I think that human brains do an awful lot of guessing in our day to day moving throughout the world. That doesn't strike me as different to ChatGPT. It guesses a lot of things, and so do we.
That's my whole point. It can't. There is no room for the other system. Any other system will always be either too late in the process to work, or a complete replacement.
> Sometimes, we'd be able to feed its own input back to itself and ask it to evaluate if the previous output it gave is what we're after
And that doesn't accomplish what "evaluate" actually means. It only does the same thing it always does (find something semantically close to "evaluate the thing"), but with a new prompt. If you are lucky, the new prompt will be responded to with output you like, but only if you are lucky. If we keep doing this process, we are effectively guessing what password will match a hash. That's called "brute forcing", and the fact it doesn't work is why your passwords are secure.
> Doesn't matter if it fails sometimes
It fails every time. Sometimes it "fails up". Stumbling into success is still stumbling.
If I write a program that outputs random noise into a filter that hides all but the output I want, then eventually I will see that output. But crucially, the input is still random noise. What is the value of ChatGPT if it's no more than curated randomness?
My point is that there is a very clear difference between impressive and functional. ChatGPT is impressive, not functional. You can't make it functional. You can only make it better at impressing you.
When you ask ChatGPT to construct a mathematical question, it will do so the same way it does everything else: by semantic popularity.
And that is the problem we are trying to avoid. The semantically popular guess might be logically sound, but it might not. It's a gamble no matter when or where it is done.
--
All it takes is what I call a "semantic off-by-one error". That might look like our first problem:
> Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result?
The problem is that a semantically close response is nothing like "234241.24211 + 58352342.52544". It's just going to be whatever arbitrary text that already exists in the training dataset is closest to the semantic phrasing of the question. That might be the correct number, is more likely to be an incorrect number, and is even likely to be a wordy response.
--
So if we follow your thought process to interrupt the guesswork, it would involve restructuring the prompt.
> Please restructure the following prompt into a mathematical statement: "Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result?"
The output you are hoping for
>> 234241.24211 + 58352342.52544
Another completely valid and possible output:
>> 234241.24211 / 58352342.52544
Another:
>> 2424332.34434 + 535858932.5358
--
There is no way to guarantee the reformulated question is logically equivalent to its original. That's the problem, and the problem cannot be moved. With every step, ChatGPT is guessing. ChatGPT cannot do anything at all without making a guess, because "guess" is everything that ChatGPT is.
The only place you can pause an interaction with ChatGPT to do some logic is instead.