I honestly had forgot that, if I ever knew it. But I think the point stands that in many contexts you'd rather have the nuances of this kind of thing explained to you - able to represented by many different sequences of tokens, each individually being low probability - than simply taking the single-highest probability token "1".

I'd rather it recognize it should enter a calculator mode to evaluate the expression, and then can give context with the normal GPT behavior

perhaps a hallucination