Sometimes I'm like...is everybody else using the same thing that I am? This is insane.
It's obvious that what we're getting are summaries. Sometimes I think I even see vestiges of the summarizing model instructions. But, that being the case, it's bizarre why Gemini's summarizer acts like a toddler, and Claude's actually lets me learn and, more importantly, stop it at its tracks when I see it's going down the wrong path.
On the other end of the spectrum, if you toy around with the open-weights models, you see the insanity that is going on in their output. I can totally believe in a scenario where claude is basically doing that and then summarizing it into something coherent, getting rid of the 10 opposing viewpoints, "aha!" and "Hmmm, but wait!", but it does it elegantly.
I've tried constantly to use other models (and I do), but Claude spoils me to no end. I keep thinking I must be holding everything wrong.
You're of course completely right. Interestingly the "summaries" have only become more diverged over time. I now believe the big 3 are barely summarizing at all any more. I deem it likely that the small models don't get the CoT at all, they just completely make up something "plausible" solely based on the user message with a little bit of previous context. Much of the time it will be the complete opposite of something I asked for, or barely related.
Your suggestion is the right one. Everyone who uses LLMs for their job should spend some time with open models like Deepseek and see what the thinking actually looks like.
I haven't even really tried to use other models in the same way, and this thread makes me feel like I don't need to.
It seems rather ironic that Anthropic givens the most logical reasoning traces, though. You'd think if they're ahead they'd either hide them or make them produce garbage (like what seems to come out of Google's models).