LLMs can't self-correct in reasoning tasks, DeepMind study finds

Re:LLMs can't self-correct in reasoning tasks, DeepMind stud... by auggierose on Thursday October 12, 2023 @12:35PM (#37859504)

Ok, that's why I asked you to give an example. I've tried a similar thing as well, and that does indeed not work very well. The reason is that you are actually asking it to compute something! And it is not very good at computing, no matter how much you explain the computation to it. But again, that's not very different from many humans. And for my use cases, this is also not much of a problem, because if I know how to compute something (and I know how to parse grammars), then I don't need to ask the AI, I can just solve it directly.

So, instead of giving it that description, give it a few examples of correct use of the grammar, see where it makes mistakes, and then add more examples / a description that would rule out these mistakes. It is really good at generalising from a few examples to the general case.

[ Parent | on HN ]

Re:LLMs can't self-correct in reasoning tasks, DeepMind stud... by roflyear on Thursday October 12, 2023 @01:33PM (#37860247)

You can always make up excuses for things, this is a simple grammar and it can't learn it.

A person can be taught to understand this grammar - the machine lacks understanding.

[ Parent | on HN ]

Re:LLMs can't self-correct in reasoning tasks, DeepMind stud... by auggierose on Thursday October 12, 2023 @02:35PM (#37861099)

@roflyear

I am not making excuses, I am explaining to you why it cannot do what you want it to do. No, it doesn't understand a grammar you give it in this form. But you can give it in a different form more suitable to its nature, and then it does understand.

But you don't seem to understand my point, yet I am not arguing you are not intelligent. You just don't understand certain things.

[ Parent | on HN ]

Re:LLMs can't self-correct in reasoning tasks, DeepMind stud... by roflyear on Thursday October 12, 2023 @10:32PM (#37865859)

Then we demonstrated it lacks the capability of general understanding.

[ Parent | on HN ]