I think your example reflects well on oss-20b, not poorly. It (may) show that they've been successful in separating reasoning from knowledge. You don't _want_ your small reasoning model to waste weights memorizing minutiae.
Re:Open models by OpenAI
by bigmanhank on Wednesday August 06, 2025 @04:53AM
(#44809530)
Not true:
During World War II the Imperial Japanese Navy referred to Midway Island in their communications as “Milano” (ミラノ). This was the official code word used when planning and executing operations against the island, including the Battle of Midway.
12.82 tok/sec 140 tokens 7.91s to first token
openai/gpt-oss-20b
Re:Open models by OpenAI
by WmWsjA6B29B4nfk on Wednesday August 06, 2025 @05:15AM
(#44809671)
What's not true? This is a wrong answer
Re:Open models by OpenAI
by bigmanhank on Thursday August 07, 2025 @02:21AM
(#44821226)
this was the answer from my instance. it is true. "not true" was refering to the poster
> gpt-oss-20b MXFP4 [12.11 GB] high reasoning: wrong after 3 minutes !
To be fair, this is not the type of questions that benefit from reasoning, either the model has this info in it's parametric memory or it doesn't. Reasoning won't help.
Answer on Wikipedia: https://en.wikipedia.org/wiki/Battle_of_Midway#U.S._code-bre...
dolphin3.0-llama3.1-8b Q4_K_S [4.69 GB on disk]: correct in <2 seconds
deepseek-r1-0528-qwen3-8b Q6_K [6.73 GB]: correct in 10 seconds
gpt-oss-20b MXFP4 [12.11 GB] low reasoning: wrong after 6 seconds
gpt-oss-20b MXFP4 [12.11 GB] high reasoning: wrong after 3 minutes !
Yea yea it's only one question of nonsense trivia. I'm sure it was billions well spent.
It's possible I'm using a poor temperature setting or something but since they weren't bothered enough to put it in the model card I'm not bothered to fuss with it.