It depends on your tests. Your integration tests should never be impacted by a refactor, or it's not a refactor (using the definition that refactors don't change interfaces). The tests need to be rerun, but they shouldn't be changed. Your unit tests and sub-integration tests (integration of multiple units but not the whole) may be impacted by some refactors, though, but that's pushing the definition of a refactor.

Your public interface to your lower level modules/classes are the only things you need to directly test. If that's your smallest unit for testing, then a refactor should also (again, in theory) not impact them because the refactor should not change that public interface, at most changing the internal (private) functions and structure which the existing tests can verify just as they verified the unit before.

If you restructure your units (split a unit into two, for instance) then you'll have to change your unit tests and possibly some integration tests if you instantiated the previous unit directly as part of the test. But that's pushing the definition of a refactor.