A phone is in almost all regards worse than using a calculator with actual buttons. Voice is even less efficient. The major reason to use a physical calculator at your desk is because it's a dedicated device that's always ready, and fast. Anecdotally, being able to use it efficiently one-handed also plays a role.
Physical calculators are for classrooms.