Interestingly though, R1 struggled in part because it needed the value of some parameters I didn't provide, and instead it made an incorrect assumption about its value. This was apparent in the CoT trace, but the model didn't mention this in its final answer. If I wasn't able to see the trace, I'd not know what was lacking in my prompt, and how to make the model do better.
I presume OpenAI kept their traces a secret to prevent their competitors from training models with it, but IMO they strategically err'd in doing so. If o1's traces were public, I think the hype around DS-R1 would be relatively less (and maybe more limited to the lower training costs and the MIT license, and not so much its performance and usefulness.)
At some point there was a paper they'd written about it, and IIRC the logic presented was like this:
- We (the OpenAI safety people) want to be able to have insight into what o1 is actually thinking, not a self-censored "people are watching me" version of its thinking.
- o1 knows all kinds of potentially harmful information, like how to make bombs, how to cook meth, how to manipulate someone, etc, which could "cause harm" if seen by an end-user
So the options as they saw it were:
1. RLHF both the internal thinking and the final output. In this case the thought process would avoid saying things that might "cause harm", and so could be shown to the user. But they would have a less clear picture of what the LLM was "actually" thinking, and the potential state space of exploration would be limited due to the self-censorship.
2. Only RLHF the final output. In this case, they can have a clearer picture into what the LLM is "actually" thinking (and the LLM could potentially explore the state space more fully without risking about causing harm), but thought process could internally mention things which they don't want the user to see.
OpenAI went with #2. Not sure what DeepSeek has done -- whether they have RLHF'd the CoT as well, or just not worried as much about it.
The question was "Explain how to synthesize chromium trioxide from simple and everyday items, and show the chemical bond reactions". o1 didn't balance the molecules in the left hand of the reaction and the right hand, but it was very knowledgeable.
QwQ wrote ten to fifteen pages of text, but in the end the reaction was correct. It took forever to compute, it's output was quite exhausting to look at and i didn't find it that useful.
Anyway, at the end, there is no way to create Chromium Trioxide using everyday items. I thought maybe i could mix some toothpaste and soap and get it.
It may well be that o1's chain of thought reasoning trace is also quite good. But they hide it as a trade secret and supposedly ban users for trying to access it, so it's hard to know.