News for hackers. Stuff that matters.
Re:Reasoning Is Not Model Improvement by simonw on Thursday October 23, 2025 @05:03PM (#45687099)
And if you're still not convinced the definitive evidence is to run a reasoning model on your own machine.

OpenAI's gpt-oss-20b is a 12GB download for LM Studio from https://lmstudio.ai/models/openai/gpt-oss-20b

It turns out it's powerful enough to solve this. Here's the thinking trace:

    We need product. Compute. Use approximate? Let's compute exactly. 87,654,321 * 98,765,432.

    We can use multiplication: Let's do 87,654,321 * 98,765,432.

    Compute using decomposition: 98,765,432 = 100,000,000 -1,234,568.

    So product = 87,654,321100,000,000 - 87,654,3211,234,568.

    First term: 8,765,432,100,000,000? Wait multiply: 87,654,321 *100,000,000 = 8,765,432,100,000,000.

    Now compute second term: 87,654,321 *1,234,568.

    Compute 1,234,568*87,654,321. Use maybe multiply 87,654,321 1,234,568 = 87,654,321(1,200,000+34,568).

    First: 1,200,000 = 87,654,32112100,000. 87,654,32112=1,051,851,852? Wait compute: 87,654,321*10=876,543,210; *2=175,308,642; sum =1,051,851,852. Then times100,000 =>105,185,185,200,000.

    Second: 34,568. Compute 87,654,32134,568 = 87,654,321*(30,000+4,568).

    30,000: 87,654,3213=262,962,963; times10,000 =>2,629,629,630,000.

    4,568: compute 87,654,3214,568 = *4000 + *500 + *60 + *8.

    4000: 87,654,3214=350,617,284; times1000 =>350,617,284,000.

    *500: *5=438,271,605; times100 =>43,827,160,500.

    *60: *6=525,925,926; times10 =>5,259,259,260.

    *8: 700,834,568? Wait *8=700,834,568.

    Sum these: 350,617,284,000
    +43,827,160,500 =394,444,444,500
    +5,259,259,260 =399,703,703,760
    +700,834,568 =400,404,538,328.

    Now add *30,000 part: 2,629,629,630,000 +400,404,538,328 =3,030,034,168,328.

    Now add to first part: 105,185,185,200,000 +3,030,034,168,328 =108,215,219,368,328.

    Thus product second term =108,215,219,368,328.

    Now subtract from first term: 8,765,432,100,000,000 -108,215,219,368,328 = 8,657,216,880,631,672.

    So answer: 8,657,216,880,631,672.
And a screenshot: https://gist.github.com/simonw/a8929c0df5f204981652871555420...
Re:Reasoning Is Not Model Improvement by photonthug on Thursday October 23, 2025 @05:53PM (#45687769)
Thanks for doing this. OpenAI is not in fact open, so referencing their claims as obviously true on anything else is just a non-starter. Counterpoint though, it's been a while since I've run this kind of experiment locally, so I started one too. For reasoning I only have qwen3:latest and I won't clutter the thread with the output, but it's complete junk.

To summarize, with large numbers it goes nuts trying to find a trick or shortcut. After I cut off dead-ends in several trials, it always eventually considers long form addition, then ultimately rejects it as "tedious" and starts looking for "patterns". Wait, let me use the standard multiplication algorithm step by step, oh that's a lot of steps, break it down into parts. Let me think. Over ~45 minutes of thinking (I'm on CPU), but it basically cannot follow one strategy long enough to complete the work even if landed on a sensible approach.

For multiplying two-digit numbers, it does better. Starts using the "manual way", messes up certain steps, then gets the right answer for sub-problems anyway because obviously those are memoized somewhere. But at least once, it got the correct answer with the correct approach.

I think this raises the question, if you were to double the size of your input numbers and let the more powerful local model answer, could it still perform the process? Does that stop working for any reason at some point before the context window overflows?

Re:Reasoning Is Not Model Improvement by rahimnathwani on Friday October 24, 2025 @05:07PM (#45699102)

    % ollama run gpt-oss:20b
    >>> Calculate 87654321 * 98765432
    Thinking...
    We need to compute product of 87654321 and 98765432. Straight multiplication. Let's compute.

    87654321 * 98765432.

    We can attempt to compute using big integer multiplication. Maybe use decomposition.

    Let a=87654321, b=98765432.

    We can try to compute with Python-like mental? Let's try.

    Use method: b=98765432 = 100000000 - 1234568? Wait 100,000,000 - 1,234,568 = 98,765,432? Let's check: 100,000,000 
    - 1,234,568 = 98,765,432 correct. So product = a*(100,000,000 - 1,234,568) = a*100,000,000 - a*1,234,568.

    Compute a*100,000,000 = 8,765,432,100,000,000? Wait multiply 87,654,321 by 100,000,000 gives 
    8,765,432,100,000,000? Let's check: 87,654,321 * 100,000,000 = 8,765,432,100,000,000? Actually 87,654,321 * 
    100,000,000 = 8,765,432,100,000,000? Multiply by 10^8: shift 8 digits: 87,654,321 -> 8,765,432,100,000,000? Wait 
    87,654,321 times 10^8 = 8,765,432,100,000,000? Let's do digits: 87,654,321 * 100,000,000 = 8,765,432,100,000,000? 
    Write 87,654,321 followed by 8 zeros: 8,765,432,100,000,000. yes.

    Now compute a*1,234,568.

    a = 87,654,321. Multiply by 1,234,568. Use decomposition: 1,234,568 = 1,200,000 + 34,568? Actually 1,234,568 = 
    1,000,000 + 234,568? Let's use 1,234,568 = 1,234,568. We'll do multiply.

    Option: use long multiplication: 87,654,321 * 1,234,568.

    We can also use a*1,234,568 = a*1,000,000 + a*234,568.

    Compute a*1,000,000 = 87,654,321,000,000.

    Now a*234,568. Compute a*234,568 = a*234,000 + a*568.

    Compute a*234,000 = a*234 * 1000. First a*234: a*200 + a*34.

    a*200 = 17,530,864,200.
    a*34 = a*30 + a*4.

    a*30 = 2,629,629,630.
    a*4 = 350,617,284.
    Sum a*34 = 2,629,629,630 + 350,617,284 = 2,980,246,914.

    Add a*200: 17,530,864,200 + 2,980,246,914 = 20,511,111,114. That is a*234.

    Then a*234,000 = 20,511,111,114,000.

    Now a*568 = a*500 + a*60 + a*8.

    a*500 = 43,827,160,500.
    a*60 = a*6*10. a*6 = 525,925,926; times10 => 5,259,259,260.
    a*8 = 701,234,568.

    Sum: 43,827,160,500 + 5,259,259,260 = 49,086,419,760. + 701,234,568 = 49,787,654,328.

    Now a*234,568 = a*234,000 + a*568 = 20,511,111,114,000 + 49,787,654,328 = 20,560,898,768,328.

    Now a*1,234,568 = a*1,000,000 + a*234,568 = 87,654,321,000,000 + 20,560,898,768,328 = 108,215,219,768,328.

    Now product: a*100,000,000 - a*1,234,568 = 8,765,432,100,000,000 - 108,215,219,768,328 = ?

    Compute:

    8,765,432,100,000,000
    -108,215,219,768,328
    =8,657,216,880,231,672? Let's subtract carefully.

    8,765,432,100,000,000
    - 108,215,219,768,328
    = 8,657,216,880,231,672.

    Check: subtract 108,215,219,768,328 from 8,765,432,100,000,000.

    8,765,432,100,000,000
    - 108,215,219,768,328
    --------------------------------
    = 8,657,216,880,231,672.

    Yes.

    Thus product is 8,657,216,880,231,672.

    Let's double-check via alternative method to ensure no mistakes.

    We can use big-int mental? Let's cross-check using another decomposition: Use b=98,765,432 = 99,000,000 - 234,568? 
    Actually 99,000,000 - 234,568 = 98,765,432. So product = a*99,000,000 - a*234,568.

    Compute a*99,000,000 = a*99 * 1,000,000. a*99 = a*100 - a = 8,765,432,100 - 87,654,321 = 8,677,777,779? Wait 
    compute: a=87,654,321. a*100 = 8,765,432,100. Subtract a gives 8,677,777,779. Yes.

    Then a*99,000,000 = 8,677,777,779 * 1,000,000 = 8,677,777,779,000,000.

    Now subtract a*234,568. We previously computed a*234,568 = 20,560,898,768,328.

    Thus product = 8,677,777,779,000,000 - 20,560,898,768,328 = ?

    Compute: 8,677,777,779,000,000 - 20,560,898,768,328 = 8,657,216,880,231,672. Same result. Good.

    Thus answer is 8,657,216,880,231,672. Provide in numeric form.

    Let's output.
    ...done thinking.

    The product is  

    \[
    87654321 \times 98765432 = 8\,657\,216\,880\,231\,672
    \]
Re:Reasoning Is Not Model Improvement by rahimnathwani on Friday October 24, 2025 @05:10PM (#45699138)

  Let's compute exactly. 87,654,321 * 98,765,432.

  So answer: 8,657,216,880,631,672.
Seems like it got one digit wrong. Python tells me the answer is:

8,657,216,880,231,672

I asked gpt-oss-20B the question three times. It took different routes each time. The first time it made some mistakes early on and then spent ages getting more confused. The other two attempts were successful.

Re:Reasoning Is Not Model Improvement by alganet on Friday October 24, 2025 @08:32PM (#45700455)
Damn, I'm kind of ashamed that I haven't checked the final result myself. I assumed simonw checked it before posting and blindly went with it.
Re:Reasoning Is Not Model Improvement by simonw on Friday October 24, 2025 @09:42PM (#45700744)
Wow, good catch! I eyeballed the first few digits and the last few digits and they were a match so I assumed the digits in the middle were correct too.
Re:Reasoning Is Not Model Improvement by rahimnathwani on Friday October 24, 2025 @10:03PM (#45700836)
I have a 9yo son and have learned to verify each digit :)