Makes sense. I guess for any given refactoring, the advantage of doing it sooner is that you get residual recurring benefits sooner. And the advantage of doing it later is you have more information about the requirements of your application and the "correct" way to do the refactor (or even if that code is going to be a long-lasting part of the app, cc Lindy effect). So whenever you touch any given piece of code which could use a refactor, you could try to figure out which advantage is more important right now.

Of course, if the total effort of doing the refactor plus building the feature on top of the refactor is less than building the feature on top of the debt, you should always do the refactor first. That's pretty much a no-brainer.

Another idea I've thought about is informally keeping track of possible refactors, maybe give them an informal prioritization, and taking a glance at your list before each task. Maybe there's some value in letting a refactor stew in your brain a little bit before actually putting it in to action. And continually maintaining a rough refactoring plan could help prevent your codebase from entering a state where it's actually unrecoverable.