News for hackers. Stuff that matters.
Re:Notes on OpenAI's new o1 chain-of-thought models by j_maffe on Friday September 13, 2024 @07:05AM (#41530016)
It does mean something. It means that the model is still more on the memorization side than being able to independently evaluate a question separate from the body of knowledge it has amassed.
Re:Notes on OpenAI's new o1 chain-of-thought models by accountnum on Friday September 13, 2024 @07:30AM (#41530172)
No, that's not a conclusion we can draw, because there is nothing much more to do than memorize the answer to this specific trick question. That's why it's a trick question, it goes against expectations and therefore the generalized intuitions you have about the domain.

We can see that it doesn't memorize much at all by simply asking other questions that do require subtle understanding and generalization.

You could ask the model to walk you through an imaginary environment, describing your actions. Or you could simply talk to it, quickly noticing that for any longer conversation it becomes impossibly unlikely to be found in the training data.

Re:Notes on OpenAI's new o1 chain-of-thought models by KoolKat23 on Friday September 13, 2024 @07:31AM (#41530177)
If you read into the thinking of the above example it wonders whether it is some sort of trick question. Hardly memorization.