Sure, once you know how to multiply you don't care about it. But try learning first year CS math without being able to multiply without perfect command of the multiplication table
Fluency in arithmetic isn't something drilled into kids just to be obnoxious, it's foundational to almost all future math skills.
Generally I agree (because the content of modern mathematics is largely abstract), but to nitpick a bit, number theory is part of mathematics too!
Ramanujan and Euler, for example, certainly cared a lot about 'arithmetic', and historically, many parts of mathematics have been just as 'empirical' in terms of calculating things as it is based on abstract proofs.
Arithmetic is the "write one sentence" of composition. The ability to think through a series of calculations with real-world context and consequences is the 5-paragraph essay. If you're not competent with the basics, you won't be able to accomplish the more advanced skill. Being tied to a calculator (not merely using, but being unable to not use) takes away intellectual agency in the same way as an LLM-generated essay (though, I'll agree, to a lesser degree).
Synthesizing an original thesis, like what people are supposed to do in writing essays, is totally different. It's a fundamental life skill people will need in all sorts of contexts, and using an LLM to do it for you takes away your intellectual agency in a way that using a calculator doesn't.