Here's OpenAI's tweet about this: https://twitter.com/SebastienBubeck/status/19465776504050567...

> Just to spell it out as clearly as possible: a next-word prediction machine (because that's really what it is here, no tools no nothing) just produced genuinely creative proofs for hard, novel math problems at a level reached only by an elite handful of pre‑college prodigies.

My notes: https://simonwillison.net/2025/Jul/19/openai-gold-medal-math...

They DID use tools for the International Collegiate Programming Contest (ICPC) programming one though: https://twitter.com/ahelkky/status/1971652614950736194

> For OpenAI, the models had access to a code execution sandbox, so they could compile and test out their solutions. That was it though; no internet access.

Given how much bad press OpenAI got just last week[1] when one one of their execs clumsily (and I would argue misleadingly) described a model achievement and then had to walk it back amid widespread headlines about their dishonesty, those researchers have a VERY strong incentive to tell the truth.

[1] https://techcrunch.com/2025/10/19/openais-embarrassing-math/

It's also worth taking professional integrity into account. Even if OpenAI's culture didn't value the truth individual researchers still care about being honest.

In OpenAI’s case, this isn’t exactly the first time they’ve been caught doing something ethically misguided:

https://techcrunch.com/2025/01/19/ai-benchmarking-organizati...

If you call the OpenAI API for o1 and ask it to multiply two large numbers it cannot use Python to help it.

Try this:

    curl https://api.openai.com/v1/responses \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $OPENAI_API_KEY" \
      -d '{
        "model": "o1",
        "input": "Multiply 87654321 × 98765432",
        "reasoning": {
          "effort": "medium",
          "summary": "detailed"
        }
      }'

o1 correctly answered the multiplication by running a long multiplication process entirely through reasoning tokens.

> "tool_choice": "auto"

> "parallel_tool_calls": true

Can you remake the API call explicitly asking it to not perform any tool calls?

Those are its default settings whether or not there are tools configured. You can set tool_choice to the name of a specific tool in order to force it to use that tool.

I added my comment here to show an example of an API call with Python enabled: https://news.ycombinator.com/item?id=45686779

Update: Looks like you can add "tool_choice": "none" to prevent even tools you have configured from being called. https://platform.openai.com/docs/api-reference/responses/cre...

Can you remake the call explicitly using the value `none`?

Maybe it's not using Python, but it's using something else. I think it's a good test. If you're right, then the response shouldn't change.

Update: `auto` is ambiguous. It doesn't say whether is picking from your selection of tools or the pool of all available tools. Explicit is better than implicit. I think you should do the call with `none`, it can't hurt and it can prove me wrong.

    curl https://api.openai.com/v1/responses \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $OPENAI_API_KEY" \
      -d '{
        "model": "o1",
        "input": "Multiply 87654321 × 98765432",
        "reasoning": {
          "effort": "medium",
          "summary": "detailed"
        },
        "tool_choice": "none"
      }'

I promise you it is not using anything else. It is performing long multiplication entirely through model reasoning.

(I suggest getting your own OpenAI API key so you can try these things yourself.)

OpenAI's gpt-oss-20b is a 12GB download for LM Studio from https://lmstudio.ai/models/openai/gpt-oss-20b

It turns out it's powerful enough to solve this. Here's the thinking trace:

    We need product. Compute. Use approximate? Let's compute exactly. 87,654,321 * 98,765,432.

    We can use multiplication: Let's do 87,654,321 * 98,765,432.

    Compute using decomposition: 98,765,432 = 100,000,000 -1,234,568.

    So product = 87,654,321100,000,000 - 87,654,3211,234,568.

    First term: 8,765,432,100,000,000? Wait multiply: 87,654,321 *100,000,000 = 8,765,432,100,000,000.

    Now compute second term: 87,654,321 *1,234,568.

    Compute 1,234,568*87,654,321. Use maybe multiply 87,654,321 1,234,568 = 87,654,321(1,200,000+34,568).

    First: 1,200,000 = 87,654,32112100,000. 87,654,32112=1,051,851,852? Wait compute: 87,654,321*10=876,543,210; *2=175,308,642; sum =1,051,851,852. Then times100,000 =>105,185,185,200,000.

    Second: 34,568. Compute 87,654,32134,568 = 87,654,321*(30,000+4,568).

    30,000: 87,654,3213=262,962,963; times10,000 =>2,629,629,630,000.

    4,568: compute 87,654,3214,568 = *4000 + *500 + *60 + *8.

    4000: 87,654,3214=350,617,284; times1000 =>350,617,284,000.

    *500: *5=438,271,605; times100 =>43,827,160,500.

    *60: *6=525,925,926; times10 =>5,259,259,260.

    *8: 700,834,568? Wait *8=700,834,568.

    Sum these: 350,617,284,000
    +43,827,160,500 =394,444,444,500
    +5,259,259,260 =399,703,703,760
    +700,834,568 =400,404,538,328.

    Now add *30,000 part: 2,629,629,630,000 +400,404,538,328 =3,030,034,168,328.

    Now add to first part: 105,185,185,200,000 +3,030,034,168,328 =108,215,219,368,328.

    Thus product second term =108,215,219,368,328.

    Now subtract from first term: 8,765,432,100,000,000 -108,215,219,368,328 = 8,657,216,880,631,672.

    So answer: 8,657,216,880,631,672.

To summarize, with large numbers it goes nuts trying to find a trick or shortcut. After I cut off dead-ends in several trials, it always eventually considers long form addition, then ultimately rejects it as "tedious" and starts looking for "patterns". Wait, let me use the standard multiplication algorithm step by step, oh that's a lot of steps, break it down into parts. Let me think. Over ~45 minutes of thinking (I'm on CPU), but it basically cannot follow one strategy long enough to complete the work even if landed on a sensible approach.

For multiplying two-digit numbers, it does better. Starts using the "manual way", messes up certain steps, then gets the right answer for sub-problems anyway because obviously those are memoized somewhere. But at least once, it got the correct answer with the correct approach.

I think this raises the question, if you were to double the size of your input numbers and let the more powerful local model answer, could it still perform the process? Does that stop working for any reason at some point before the context window overflows?

    % ollama run gpt-oss:20b
    >>> Calculate 87654321 * 98765432
    Thinking...
    We need to compute product of 87654321 and 98765432. Straight multiplication. Let's compute.

    87654321 * 98765432.

    We can attempt to compute using big integer multiplication. Maybe use decomposition.

    Let a=87654321, b=98765432.

    We can try to compute with Python-like mental? Let's try.

    Use method: b=98765432 = 100000000 - 1234568? Wait 100,000,000 - 1,234,568 = 98,765,432? Let's check: 100,000,000 
    - 1,234,568 = 98,765,432 correct. So product = a*(100,000,000 - 1,234,568) = a*100,000,000 - a*1,234,568.

    Compute a*100,000,000 = 8,765,432,100,000,000? Wait multiply 87,654,321 by 100,000,000 gives 
    8,765,432,100,000,000? Let's check: 87,654,321 * 100,000,000 = 8,765,432,100,000,000? Actually 87,654,321 * 
    100,000,000 = 8,765,432,100,000,000? Multiply by 10^8: shift 8 digits: 87,654,321 -> 8,765,432,100,000,000? Wait 
    87,654,321 times 10^8 = 8,765,432,100,000,000? Let's do digits: 87,654,321 * 100,000,000 = 8,765,432,100,000,000? 
    Write 87,654,321 followed by 8 zeros: 8,765,432,100,000,000. yes.

    Now compute a*1,234,568.

    a = 87,654,321. Multiply by 1,234,568. Use decomposition: 1,234,568 = 1,200,000 + 34,568? Actually 1,234,568 = 
    1,000,000 + 234,568? Let's use 1,234,568 = 1,234,568. We'll do multiply.

    Option: use long multiplication: 87,654,321 * 1,234,568.

    We can also use a*1,234,568 = a*1,000,000 + a*234,568.

    Compute a*1,000,000 = 87,654,321,000,000.

    Now a*234,568. Compute a*234,568 = a*234,000 + a*568.

    Compute a*234,000 = a*234 * 1000. First a*234: a*200 + a*34.

    a*200 = 17,530,864,200.
    a*34 = a*30 + a*4.

    a*30 = 2,629,629,630.
    a*4 = 350,617,284.
    Sum a*34 = 2,629,629,630 + 350,617,284 = 2,980,246,914.

    Add a*200: 17,530,864,200 + 2,980,246,914 = 20,511,111,114. That is a*234.

    Then a*234,000 = 20,511,111,114,000.

    Now a*568 = a*500 + a*60 + a*8.

    a*500 = 43,827,160,500.
    a*60 = a*6*10. a*6 = 525,925,926; times10 => 5,259,259,260.
    a*8 = 701,234,568.

    Sum: 43,827,160,500 + 5,259,259,260 = 49,086,419,760. + 701,234,568 = 49,787,654,328.

    Now a*234,568 = a*234,000 + a*568 = 20,511,111,114,000 + 49,787,654,328 = 20,560,898,768,328.

    Now a*1,234,568 = a*1,000,000 + a*234,568 = 87,654,321,000,000 + 20,560,898,768,328 = 108,215,219,768,328.

    Now product: a*100,000,000 - a*1,234,568 = 8,765,432,100,000,000 - 108,215,219,768,328 = ?

    Compute:

    8,765,432,100,000,000
    -108,215,219,768,328
    =8,657,216,880,231,672? Let's subtract carefully.

    8,765,432,100,000,000
    - 108,215,219,768,328
    = 8,657,216,880,231,672.

    Check: subtract 108,215,219,768,328 from 8,765,432,100,000,000.

    8,765,432,100,000,000
    - 108,215,219,768,328
    --------------------------------
    = 8,657,216,880,231,672.

    Yes.

    Thus product is 8,657,216,880,231,672.

    Let's double-check via alternative method to ensure no mistakes.

    We can use big-int mental? Let's cross-check using another decomposition: Use b=98,765,432 = 99,000,000 - 234,568? 
    Actually 99,000,000 - 234,568 = 98,765,432. So product = a*99,000,000 - a*234,568.

    Compute a*99,000,000 = a*99 * 1,000,000. a*99 = a*100 - a = 8,765,432,100 - 87,654,321 = 8,677,777,779? Wait 
    compute: a=87,654,321. a*100 = 8,765,432,100. Subtract a gives 8,677,777,779. Yes.

    Then a*99,000,000 = 8,677,777,779 * 1,000,000 = 8,677,777,779,000,000.

    Now subtract a*234,568. We previously computed a*234,568 = 20,560,898,768,328.

    Thus product = 8,677,777,779,000,000 - 20,560,898,768,328 = ?

    Compute: 8,677,777,779,000,000 - 20,560,898,768,328 = 8,657,216,880,231,672. Same result. Good.

    Thus answer is 8,657,216,880,231,672. Provide in numeric form.

    Let's output.
    ...done thinking.

    The product is  

    \[
    87654321 \times 98765432 = 8\,657\,216\,880\,231\,672
    \]

  Let's compute exactly. 87,654,321 * 98,765,432.

  So answer: 8,657,216,880,631,672.

8,657,216,880,231,672

I asked gpt-oss-20B the question three times. It took different routes each time. The first time it made some mistakes early on and then spent ages getting more confused. The other two attempts were successful.

I can see this call now has a lot more tokens for the reasoning steps. Maybe that's normal variance though.

(I don't have a particular interest in proving or disproving LLM things, so there's no incentive for me to get a key). There was an ambiguous point in the "proof", I just highlighted it.

You can also get an account with something like https://openrouter.ai/ which gives you one key to use with multiple different backends.

Or use GitHub Models which gives you free albeit limited access to a bunch at once. https://github.com/marketplace/models

Lots of people don't have resources to invest in LLMs (either self-hosted or not). They rely on what other people say. And people get caught in the hype all the time. As it turns out, lots of hype nowadays is around LLMs, so that's where I'll go.

I was skeptic about LK99. Didn't had the resources to independently verify it. It doesn't mean I don't believe in superconductors or that I should have no say in it.

Some of that hype will be justified, some will not. And that's exactly what I expect from this kind of technology.

I can invest lots of time in Linux, for example. I don't know how to write a driver for it, but I know I could learn how to do it. If there's a bug in a driver, there's nothing stopping me except my own will to learn. I can also do it in a potato, or my phone.

I can experiment with free tier LLMs, but that's as far as I will go. It's not just about me, that is as far as 99% of the developers will go.

So, it's not uninteresting because it's boring or something. It's uninteresting because it puts a price on learning. That horizon of "if there's a bug in it, I can fix it" is severely limited. That's a price most free software developers are not considering worthy. There's a lot of us.

I love learning about software. That's why I'm leaning so heavily on LLMs these days - they let me learn so much faster, and let me dig into whole new areas that previously I would never have considered experimenting with.

Just this week LLMs helped me figure out how to run Perl inside WebAssembly in a browser... and then how to compile 25-year-old C code to run in WebAssembly in the browser too. https://simonwillison.net/2025/Oct/22/sloccount-in-webassemb...

If I'd done this without LLMs I might have learned more of the underlying details... but realistically I wouldn't have done this at all, because my interest in Perl and C in WebAssembly is not strong enough to justify investing more than a few hours of effort.

A while back, I didn't even knew those problems existed. It took me a while to understand them and why they're interesting and lots of people spend time on them.

I have tried to adapt the problems to the LLMs as well, such as shaping the problem to look more like a thing that they're alreay trained on, but I soon realized the limitations of that approach.

I think in a couple of decades, maybe earlier, that kind of thing will be commonplace. People training their own stuff from scratch, on cheap hardware. It will unleash an even more rewarding learning experience for those willing to go the extra mile.

I think you're missing that perspective. That's fine, by the way. You're totally cool and probably helping lots of people with your work. I support it, it allows people to understand better where LLMs currently can help and where they cannot.

One of the reasons I am so excited about the "skills" concept from Anthropic is that it helps emphasize how the latest generation of LLMs really can pick up new capabilities if you get them to read a single, carefully constructed markdown file.

https://github.com/fosslinux/live-bootstrap/

Other efforts around the same problem are trying to make it more architecture independent or improve regenerations (re-building things like automake during the process).

It's free and open source, you're welcome to fork it and try your best with the aid of Claude. All you need is an x86 or x86-64 machine or qemu.

The project and other related repositories are already full of documentation in the markdown format and high quality commented code.

Here's a friendly primer on the problem:

https://www.youtube.com/watch?v=Fu3laL5VYdM

If you decide to help, please ask the maintainers if AI use is allowed beforehand. I'm OK with it, they might not be.

    curl https://api.openai.com/v1/responses \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $OPENAI_API_KEY" \
      -d '{
        "model": "gpt-5",
        "input": "Multiply 87654321 × 98765432",
        "reasoning": {
          "effort": "medium",
          "summary": "detailed"
        },
        "tools": [
          {
            "type": "code_interpreter",
            "container": {"type": "auto"}
          }
        ]
      }'

Note this bit where the code interpreter Python tool is called:

    {
      "id": "rs_080a5801ca14ad990068fa91f2779081a0ad166ee263153d98",
      "type": "reasoning",
      "summary": [
        {
          "type": "summary_text",
          "text": "**Calculating large product**\n\nI see that I need to compute the product of two large numbers, which involves big integer multiplication. It\u2019s a straightforward task, and given that I can use Python, that seems like the best route to avoid any potential errors. The user specifically asked for this multiplication, so I\u2019ll go ahead and use the python tool for accurate analysis. Let\u2019s get started on that!"
        }
      ]
    },
    {
      "id": "ci_080a5801ca14ad990068fa91f4dbe481a09eb646af049541c6",
      "type": "code_interpreter_call",
      "status": "completed",
      "code": "a = 87654321\r\nb = 98765432\r\na*b",
      "container_id": "cntr_68fa91f12f008191a359f1eeaed561290c438cc21b3fc083",
      "outputs": null
    }

1. You can enable or disable tool use in most APIs. Generally, tools such as web search and Python interpreter give models an edge. The same is true for humans, so, no surprise. At the frontier, model performance keeps climbing - both with tool use enabled and with it disabled.

2. Model capabilities keep improving. Frontier models of today are both more capable at their peak, and pack more punch for their weight. Capability per trained model weight and capability per unit of inference compute are both rising. This is reflected directly in model pricing - "GPT-4 level of performance" is getting cheaper over time.

3. We're 3 years into the AI revolution. If I had ten bucks for every "breakthrough new architecture idea" I've seen in a meanwhile, I'd be able to buy a full GB200 NVL72 with that.

As a rule: those "breakthroughs" aren't that. At best, they offer some incremental or area-specific improvements that could find their way into frontier models eventually. Think +4% performance across the board, or +30% to usable context length for the same amount of inference memory/compute, or a full generational leap but only in challenging image understanding tasks. There are some promising hybrid approaches, but none that do away with "autoregressive transformer with attention" altogether. So if you want a shiny new architecture to appear out of nowhere and bail you out of transformer woes? Prepare to be disappointed.

The original question still stands: do recent LLMs have an inherent knowledge of arithmetic, or do they have to offload the calculation to some other non-LLM system?

[0] https://www.mindprison.cc/p/why-llms-dont-ask-for-calculator...

Which includes, among other things, the underappreciated metacognitive skill of "being able to decide when to do math quick and dirty, in one forward pass, and when to write it out explicitly and solve it step by step".

Today's frontier LLMs can do that. A lot of training for "reasoning" is just training for "execute on your knowledge reliably". They usually can solve math problems with no tool calls. But they will tool call for more complex math when given an option to.

Which was also when the capabilities of LLMs became completely impossible to either ignore or excuse as "just matching seen data". But that was, in practice, solvable simply by increasing the copium intake.

This improved think->act->sense loop that they now form, exponentially increases the possible utility of the models. We are just starting to see this with gpt-5 and the 4+ series of Claude models.

Caveat that we don't fully understand how human intelligence works, but with humans it's generally true that skills are not static or siloed. Improving in one area can generate dividends in others. It's like how some professional football players improve their games by taking ballet lessons. Two very different skills, but the incorporation of one improves the other as well as the whole.

I would argue that narrowly focusing on LLM performance via benchmarks before tool use is incorporated is interesting, but not particularly relevant to whether they are transformative, or even useful, as products.

Reasoning just means more implicit chain-of-thought. It can be emulated by non reasoning model by explicitly constructing prompt to perform longer step by step thought process. With reasoning models it just happens implicitly, some models allow for control over reasoning effort with special tokens.

Nothing plateaued from where I'm standing – new models are being trained, releases happen frequently with impressive integration speed. New models outperform previous ones. Models gain multi modality etc.

Regarding alternative architectures – there are new ones proposed all the time. It's not easy to verify all of them at scale. Some ideas that are extending current state of art architectures end up in frontier models - but it takes time to train so lag does exist. There are also a lot of improvements that are hidden from public by commercial companies.

Both reasoning and non-reasoning models may choose to use the Python interpreter to solve math problems. This isn't hidden from the user; it will show the interpreter ("Analyzing...") and you can click on it to see the code it ran.

It can also solve math problems by working through them step-by-step. In this case it will do long multiplication using the pencil-and-paper method, and it will show its work.

More broadly I think what we’re looking for at the end of the day, AGI, is going come about from a diaspora of methods capturing the diverse aspects of what we recognize as intelligence. ‘Precise deductive reasoning’ is one capability out of many. Attention isn’t all you need, neither is compression, convex programming, what have you. The perceived “smoothness” or “unity” of our intelligence is an illusion like virtual memory hiding cache, and building it is going to look a lot more like stitching these capabilities together than deriving some deep and elegant equation.