The code is where TODO's should live. Anything else can be separated from it. It'll help people understand areas where technical debt was intentionally incurred and expose them to the tradeoffs involved at a high-level.

Also, not all tech debt is solved by refactoring.

TODO comments aren't suitable for tracking all technical debt, but they're good for targeted instances.