Debugging is reverse-engineering under the gun. It has huge cognitive load. Especially debugging a production system with a difficult to reproduce bug in a deep dark part of the code. It's a nightmare scenario.
Refactoring, on the other hand, often happens with incomplete knowledge, and can be quite local. I've seen zillions of refactorings that are done with incomplete knowledge that are local improvements (and many that were not global improvements).
When I say non-trivial, I don't mean local refactoring. I mean the kind of refactoring that requires you to load the entire system (or a large part of it) into your head, and figure out how to clarify and simplify it.
It is not a luxury. When done successfully, it is the only way to lower the cost of that expensive debugging. The slow debugging and the expensive refactoring are two sides of the same coin. They are both the cost of a system that is too difficult to understand and safely change. But the cost of a good refactoring need only be paid once. Whereas the cost of debugging a system you refuse to fix is levied again and again.
No matter how many you have, they'll only be testing a tiny fraction of your possible code paths.
Refactoring is by far the most expensive and error prone activity in programming. It can also be one of the most valuable. But unless it's trivial, it's the most mentally arduous and time-consuming work you do as a programmer.