And I think you’re right when you say that they’re lacking in recursive thinking abilities. However, their reasoning abilities are pretty excellent which is why when you prompt them to think step-by-step, or break down problems to them, they correctly output the right answer.
If you're solving a complex problem, you cannot expect it to "reason" about it. You have to break the problem into simpler pieces, then you can have the LLM do the grunt work for each piece.
Are there any datasets out there that provide the full edit stream of a human from idea to final refinement, that a model could be trained on?
Other good examples narratives that include a lot of internal monologue. Thing a book written in the form:
> The sphinx asked him, "A ham sandwich costs $1.10. The ham costs $1 more than the bread. How much does the bread cost?"
> He thought carefully. He knew the sphinx asked tricky problems. If the ham costs a dollar more than the bread, the bread couldn't possibly be more than 10 cents. But if the bread was 10 cents, the ham would be $1.10 and the total would be $1.20. That can't be. We need to lose 10 cents, and it has to be divided evenly among the ham and bread to maintain the dollar offset. So the ham must be $1.05 and the bread must be $0.05. He answered the sphinx confidentally "The bread is $0.05!".
Don't think of an LLM as a full "computer" or "brain". Think of it like a CPU. Your CPU can't run whole programs, it runs single instructions. The rest of the computer built around the CPU gives it the ability to run programs.
Think of the LLM like a neural CPU whose instructions are relatively simple English commands. Wrap the LLM in a script that executes commands in a recursive fashion.
Yes, you can get the LLM to do complicated things in a single pass, this is a testament to the sheer size and massive training set of GPT3 and its ilk. But even with GPT3 you will have more success with wrapper programs structured like:
This program breaks down the task of writing a good premise into a cycle of writing/critique/rewriting. You will get better premises this way than if you just expect the model to output one on the first go.You can somewhat emulate a few layers of this without wrapper code by giving it a sequence of commands, like "Write a movie premise, then write a critique of the movie premise, then rewrite the premise taking into account the critique".
The model is just trained to take in some text and predict the next word (token, really, but same idea). Its training data is a copy of a large swath of the internet. When humans write, they have the advantage of thinking in a recursive fashion offline, then writing. They often edit and rewrite before posting. GPT's training process can't see any of this out-of-text process.
This is why it's not great at logical reasoning problems without careful prompting. Humans tend to write text in the format "<thesis/conclusion statement><supporting arguments>". So GPT, being trained on human writing, is trained to emit a conclusion first. But humans don't think this way, they just write this way. But GPT doesn't have the advantage of offline thinking. So it often will state bullshit conclusions first, and then conjure up supporting arguments for it.
GPT's output is like if you ask a human to start writing without the ability to press the backspace key. It doesn't even have a cognitive idea that such a process exists due to its architecture and training.
To extract best results, you have to bolt on this "recursive thinking process" manually. For simple problems, you can do this without a wrapper script with just careful prompting. I.e. for math/logic problems, tell it solve the problem and show its work along the way. It will do better since this forces it to "think through" the problem rather than just stating a conclusion first.