> But the PM doesn't understand why you're changing the code you made
If that is the case, the PM does not understand red-green-refactor, always-refactoring, or even just plain refactoring. I'd say the PM doesn't even understand the idea of technical debt, in that case.
The problem is certainly the PM's disconnect to how to keep a codebase (and their team) sane in the most common programming practices. And not the refactoring itself.
1. You write some code.
2. You write a duplicate of #1.
3. You write another duplicate of #1, so you should refactor. But the PM doesn't understand why you're changing the code you made in #1 and #2 as part of ticket #3.
So the jaded devs are certain the duplication will occur, but believe their choice is either to pre-emptively fix the duplication at #1, or never to fix it at all.