(I only RTFA after writing this comment, and I now see that the below is what they're doing)
I'm an outsider to this field. My unexpert thought was that perhaps this model could be used to identify the maths components of that question and then we (programmatically) feed that into a different system that gives the answer. That answer then could be provided as context when asking the real question, so the model has access to the mathematical facts.
E.g., I just put this question into the playground:
"Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result?"
Identify the mathematical question in the preceding question and write it as an equation.
234241.24211 + 58352342.52544 = ?
We take the result of that sum from a separate system, and redo the same question, providing the mathematical details as context. E.g., now asking:
234241.24211 + 58352342.52544 = 58586583.76755
Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result?
The result is 58586583.76755.
6,179,563.76754When you ask ChatGPT to construct a mathematical question, it will do so the same way it does everything else: by semantic popularity.
And that is the problem we are trying to avoid. The semantically popular guess might be logically sound, but it might not. It's a gamble no matter when or where it is done.
--
All it takes is what I call a "semantic off-by-one error". That might look like our first problem:
> Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result?
The problem is that a semantically close response is nothing like "234241.24211 + 58352342.52544". It's just going to be whatever arbitrary text that already exists in the training dataset is closest to the semantic phrasing of the question. That might be the correct number, is more likely to be an incorrect number, and is even likely to be a wordy response.
--
So if we follow your thought process to interrupt the guesswork, it would involve restructuring the prompt.
> Please restructure the following prompt into a mathematical statement: "Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result?"
The output you are hoping for
>> 234241.24211 + 58352342.52544
Another completely valid and possible output:
>> 234241.24211 / 58352342.52544
Another:
>> 2424332.34434 + 535858932.5358
--
There is no way to guarantee the reformulated question is logically equivalent to its original. That's the problem, and the problem cannot be moved. With every step, ChatGPT is guessing. ChatGPT cannot do anything at all without making a guess, because "guess" is everything that ChatGPT is.
The only place you can pause an interaction with ChatGPT to do some logic is instead.
My understanding of machine learning in general is that this is not how it works, and rather it uses a neural network for a lot of what it does (which isn't merely picking the closest arbitrary text in its training set), though I don't know details about NLP specifically. I am aware that some activities involve ranking the similarity of particular texts, and picking the closest matches (that's how embeddings get used for providing context to prompts), but I didn't think that's how the underlying models operate when, say, asked the question I asked it.
Regarding the rest of your remarks, the aim isn't necessarily perfection. You listed some other possible outputs, but the fact is that it didn't offer these options, and instead it offered the correct one, and it seems that a system that uses ChatGPT + something else can provide an improvement over ChatGPT on its own (with the downside being that it requires more resources to provide those answers).
Is the other "completely valid and possible output" you provided actually likely? Even a human could possibly make the mistakes you list, but possible doesn't mean probable, and being possible doesn't take away from the results of such a hybrid system being an improvement over ChatGPT on its own.
I didn't think guarantees are the game here. Achieving something closer to what humans are (and even some humans struggle with logic and mathematics), and even exceeding it in some places, would be of immense value.
> ChatGPT cannot do anything at all without making a guess, because "guess" is everything that ChatGPT is
Is this different to what humans do? There are accounts of the brain where it takes sensory input, updates an internal model, and makes predictions (guesses) about what inputs it expects to receive. Getting better at guessing might be the right game to play. i.e., it's not a criticism to say it's guessing, because that's what these models are trying to do. If it can guess reasonably well when a question is mathematical or logical, and can guess reasonably well about how to structure a mathematical or logical equation or statement based on that text, then that could be fed into another system to produce a more accurate answer.
...because you gave it a "correct" prompt; or in other words, you gave it a prompt that ChatGPT can respond "successfully" to. But if you keep trying, you will surely find yourself with "failed" responses. And ChatGPT has no way of knowing the difference. That's my point.
> Is the other "completely valid and possible output" you provided actually likely?
We're splitting hairs here. I didn't go and test a bunch of prompts to find an example failure. You are welcome to do that if you want. It's not very hard, especially when you start adding variables. This is well-known behavior that ChatGPT itself warns its users about.
> > ChatGPT cannot do anything at all without making a guess, because "guess" is everything that ChatGPT is
> Is this different to what humans do?
Yes!
We don't just spit out the first thing that comes to mind, unless that's the task we are intending to do - and anyone who has played music can tell you just how difficult that task is.
Most of the time, we conceptualize. We comprehend the meaning of language - sure, it takes some guesswork along the way, but that's not the main event - then we construct a new idea, then we express it.
We don't just want accurate answers. We want conceptualization. That's the feature we are all dreaming about with Natural Language Processors. If we can teach a computer to conceptualize language, then we can get it to interface with any arbitrary human knowledge!
But ChatGPT doesn't interface. It parrots; and that's an insult to the birds themselves - unlike ChatGPT, they have an intelligence of their own.
The only way ChatGPT could give you a valid response about any arbitrary concept is to write every thing from every angle about every concept ever, and feed it all to ChatGPT as training data. That's exactly the work we want to avoid!
I was disagreeing with this, and showing ways it can advance. Your reply focuses on the fact that it will still get things wrong. However, I never claimed that it wouldn't make mistakes. I agree it will, and frequently. I just wanted to show that it can do more, and valuably and recognisably more.
The next step up for ChatGPT isn't perfection of mathematics and logic. And with the examples we've been playing with, experimentations with prompts, and further training, when combined with other systems, I don't see why better couldn't be done, and I gave one such example. Sometimes the output it gives is what we're after, sometimes it isn't. Sometimes, we'd be able to feed its own input back to itself and ask it to evaluate if the previous output it gave is what we're after. E.g., I recall hearing about someone feeding the output to the Rust compiler, then feed error messages back, in an iterative loop until they get something that at least compiles. I don't know the best techniques yet, but I have no doubt that a variety of techniques combined with other systems could improve the output of ChatGPT, significantly.
Regarding fair prompts, here's another example of a prompt and its reply, trying at a more generic prompt:
Provide a list of mathematical questions contained in the following text, phrased as equations: "Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result? An answer was given by Suzie, who had recently had to count how many new potatoes she had in her pile, after putting the seven new ones in the pile of 15 she already had."
1. 234241.24211 + 58352342.52544 = ___________
2. 15 + 7 = ___________
Assuming the following text is given to students, create a list of mathematical equations that could serve as questions based on this text: "Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result? An answer was given by Suzie, who had recently had to count how many new potatoes she had in her pile, after putting the seven new ones in the pile of 15 she already had."
1. 234241.24211 + 58352342.52544 = __?
2. 7 + 15 = __?
You might be right about your claims about conceptualising, and seemingly focusing on perfection as the next step. Perfection isn't the next step in this journey, because there's a lot more value to be added before trying to hit anything remotely close to that (if we even can!). (here, I'm referring to your comment: "I don't see any way to seed such a system to always produce logically correct results")
Side note: I think that humans (and probably many other animals) are conscious, and have a non-physical mind, while computers never will, or at least won't in virtue of us improving the physical characteristics of these systems. But that being said, I think there's a lot about the way human brains work that can be mimicked by machines, and I think that human brains do an awful lot of guessing in our day to day moving throughout the world. That doesn't strike me as different to ChatGPT. It guesses a lot of things, and so do we.
That's my whole point. It can't. There is no room for the other system. Any other system will always be either too late in the process to work, or a complete replacement.
> Sometimes, we'd be able to feed its own input back to itself and ask it to evaluate if the previous output it gave is what we're after
And that doesn't accomplish what "evaluate" actually means. It only does the same thing it always does (find something semantically close to "evaluate the thing"), but with a new prompt. If you are lucky, the new prompt will be responded to with output you like, but only if you are lucky. If we keep doing this process, we are effectively guessing what password will match a hash. That's called "brute forcing", and the fact it doesn't work is why your passwords are secure.
> Doesn't matter if it fails sometimes
It fails every time. Sometimes it "fails up". Stumbling into success is still stumbling.
If I write a program that outputs random noise into a filter that hides all but the output I want, then eventually I will see that output. But crucially, the input is still random noise. What is the value of ChatGPT if it's no more than curated randomness?
My point is that there is a very clear difference between impressive and functional. ChatGPT is impressive, not functional. You can't make it functional. You can only make it better at impressing you.
Right now ChatGPT acts like pure Amygdala. You poke it, and it spits out something that looks right based on what it was trained on. This is fine, it's just a demo, and this gives a really deep understanding of the direct limitations of these sorts of LLMs.
Ideally a more mature chat bot would just do what I do in situations where some critical thinking or research is called for. I do a quick fact check. I think up a few search terms for Google, Wikipedia, Wolfram Alpha, whatever, assume those results are more logically accurate, and present that data instead of the typical vomit my brain usually spits out. ChatGPT is actually really good at coming up with good search terms for popular search engines, and incorporating results into responses. It just doesn't have anything that reaches out to the web at the moment, which is probably so a certain subsection of the population doesn't have a total panic attack.
It seems like fact check functionality would take less than 100 lines of relatively straight-forward python. I bet you could even get ChatGPT to write it.
function critical_filter(response) => Maybe Text {
if logic_parse_check(response) == LogicallySound then {
return Some response
} else {
return Nothing
}
}
do {
static seed = 0;
seed += 1;
response = critical_filter(GPT::generate_response(prompt, seed))
} while response != Nothing
The only solution is a replacement.
As for the speculation itself, e.g., “as soon as you merge two subjects, you are right back to gambling semantics as logic”, it may be suggestive to consider that such models have learned to translate between English and French after having only trained on the two languages separately.
If you were to consider carefully enough the language example that I gave, you would see that it already refutes your speculation.
Another way would be to apply your argument to humans, who have managed to be useful despite having been trained only on (a broader class of) semantics (i.e., the sum of all qualia) and the logical imperfections that this entails (cf. your comments).
Happy new year!
I haven't spent much time learning about how ChatGPT works, but:
'ChatGPT does not simply output the “most semantically popular result”. That description applies only to the base model, before instruction tuning and RLHF.'
was exactly the explanation I needed!
…Unless this whole comment was generated by ChatGPT. (I strongly dislike that I’m starting to second guess whether comments are written by humans or not)
If we trained it on gibberish and nonsense, we would get that, and no one would care. The tricky part is that we don't have much gibberish or nonsense to train with: because of that pesky property of human expression we call logic.
But ultimately I think it's just too computationally heavy to do math, remember obscure facts, or track rapidly changing data, with an LLM, and it's far more efficient to pass it off to a piece of specialized software, just the same as humans do.
...but what we just described isn't a language processor anymore: it's just a database. There is no need to use a neural network if you already have a known response to every possible input!
The entire point of ChatGPT in the first place is to generate a response that doesn't exist in the training data. If all we needed was to look up the answer we wrote down earlier, we already have SQL.
What we are talking about here is being able to provide a logically correct response without just pulling it out of a lookup table. Either you can do that, or you can't, and ChatGPT can't.
My point here is even if it only "gets" the semantics, it has the ability to perform logic. It's just not very efficient. And, I'd say this isn't that far off from what is happening in our brains.
Do we really "get" logic, or do we rely on heuristics? Why do we know that 12 * 2 is equal to 24? Either because we remember our multiplication tables (look up from a table) or because we break it into smaller steps until we're left with pieces of the problem that we inherently know (including 12 * 2 means 12 + 12 which means take 12 and increment the number +1 for 12x times, or 10 + 10 = 20 and 2 + 2 = 4 so 12 + 12 = 24).
I don't see why that couldn't reasonably scaled up to advanced calculus.
I think the point that you're hitting on is that LLMs aren't "full brains" that have all of the components that human brains have, and that's true. But LLMs appear to essentially have the ability to simulate (or replicate depending on how far you want to go) executive function. As others have hit on, from that, you can either command other specialized components to be able to do specific tasks (like recall facts, perform logic, etc) or you can have it break down the problem into steps. Then take the output from that processing and form a coherent response.
The structure here is flexible enough that I'm struggling to find the upper limit, and if all research in the development of LLMs stopped tomorrow, and everyone focused on building with LLMs, I still don't think that limit will be found.
ChatGPT doesn't "know" anything, its just a prediction engine. if you ask it what 2+2 is, it will tell you 4, but it didnt actually do math. it only said 4 because everyone else said it.
there is no generalizable reasoning going on there.
ChatGPT interfaces with semantics, and not logic.
--
That means that any emergent behavior that appears logically sound is only an artifact of the logical soundness of its training data. It can only echo reason.
The trouble is, it can't choose which reason to echo! The entire purpose of ChatGPT is to disambiguate, but it will always do so by choosing the most semantically popular result.
It just so happens that the overwhelming majority of semantic relationships also happen to be logical relationships. That's an emergent effect of the fact that we are usually using words to express logic. So if you mimic human speech well enough to look semantically interesting, you are guaranteed to also appear logically sound.
--
I don't see any way to seed such a system to always produce logically correct results. You could feed it every correct statement about every subject, but as soon as you merge two subjects, you are right back to gambling semantics as logic.
I also don't see a scalable way to filter the output to be logically sound every time, because that would be like brute-forcing a hash table.
OP considers something in the middle, but that's still pretty messy. They essentially want a dialogue between ChatGPT and WolphramAlpha, but that depends entirely on how logically sound the questions generated by ChatGPT are, before they are sent to WolphramAlpha. It also depends on how capable WolphramAlpha was at parsing them.
But we already know that ChatGPT is prone to semantic off-by-one errors, so we already know that ChatGPT is incapable of generating logically sound questions.
--
As I see it, there is clearly no way to advance ChatGPT into anything more than it is today. Impressive as it is, the curtain is wide open for all to see, and the art can be viewed plainly as what it truly is: magic, and nothing more.