Some nice charts here [0], which IMO means LLMs are getting very good at guessing answers to certain arithmetic operations, but they don't actually perform it in a logical fashion.

[0] https://www.mindprison.cc/p/why-llms-dont-ask-for-calculator...