I don't think I agree with your premise, but even if I take it as true--- it's the unit tests of the other modules that help me out, by ensuring I haven't managed to totally bust the module I'm working on.
Some people argue that a unit test of a module should mock out its dependencies and not rely on them. For unit tests like that... well, I endorse the article.
Unit tests are supposed to work with the internals of a module and not just its boundaries. Since refactoring normally tries to significantly change the internals without changing the boundary, it usually also results in a rewrite of the unit tests. Integration tests are the ones that usually help me sleep better after a refactor.