This, exactly. You should always be looking to improve code. If you have a truly comprehensive test suite, there is no danger in always improving code. It's easier to make refactoring continuous than it is to figure out a checklist of when to do it. The answer is "always".