Ultimately computers are just doing lots of arithmetic.
I think ultimately it means that "doing arithmetic / doing math" are defined by the observer rather than the doer. True for any abstraction - the brain does exactly what it does, then to ask if its doing "abstracted task X" is not really meaningful from the perspective of brain, the abstraction in the first place is a mental construct we create.
Otherwise you're asserting "doing arithmetic" as a property of any analogue system, and would say that a falling ball needs to understand calculus in order to make an arc.
The brain is doing purposeful data processing. Yes it's just signals doing their thing following the laws of physics but that's also what computers do.
Arithmetics in a CPU are just electrical signals passing through wires. If you were to look at these signals without any awareness of why they are flowing this way here and there, you wouldn't be able to tell that these signals are an encoding of some arithmetic operations.
If processing audio and visual signals does not involve doing arithmetics, then what does it involve?
If people are going to argue that inanimate objects perform arithmetic, they will need to define arithmetic. Digital or analog calculators do it symbolically, subject to interpretation of the symbols by a human.
If a human is using a calculator or an abacus or blackboard to perform arithmetic, would we also say that the abacus or blackboard performs arithmetic? (Great, now we have to define "performs")
The way I see it, an operational amplifier, or a person catching a ball (to take an example from another post here) is performing a task that can be analyzed mathematically and modeled arithmetically, which is not the same as doing the analysis or setting up the model and performing the calculations.
In short, computers are good at rapidly performing the calculations of a solved problem, or implementation of the algorithm. Humans are good at coming up with the algorithm. A good, general purpose algorithm generator is probably ai-complete, although i could see "evolution" as a plausible answer to such an algorithm generator too, though maybe only in its capacity to generate beter generators. Or maybe I'm talking out my excretorial orifice, I'm often not sure.