Arbitrarily precise is not the same as exact. There are many possible examples to illustrate why this matters. Here's one: let's say that you compute floor(f(x)) where f(x) falls right on an integer n. If you evaluate f(x) in floating-point arithmetic, you probably get n +/- epsilon and not n exactly because of rounding errors. This makes the floor function go to either n or n - 1, so you sometimes end up with an error of 1 no matter how precise the floating-point numbers.
What kind of lousy CAS does not support arbitrarily precise numbers? For example Fricas/Axiom/Scratchpad gives arbitrarily precise results.