If you don't have either a united team or managers willing to move at all, or even work with ICs collaboratively, you have bigger problems than what you do with TODO et al. If I were in such a situation I'd still use an 'issue tracker' (maybe a local text file) 9 times out of 10, while also looking for a better job.
Issue trackers are slow and I have to work with one of the slowest (it was built in-house on top of a core product never meant for such a thing...) but some are much faster than others. Command line ones can be pretty slick. Still, it comes down to team (and individual) preference and experience (your "very hard"s are never even "hard" in my experience). Even on a personal project that's going to span a decent length of time, I'll take the slowness of adding even a minor issue and the rare risk of grooming/popping the backlog (and all this might just be in a single local text file, very low overhead) only to discover an item no longer applies and taking the second to Never it. It's not only faster for me in the long run but maximizes the incentives for the things I want in a system that I'm tasked with maintaining and improving.
At many companies, issue trackers are not used for things like "refactor this method". They're used for user-facing stories or architectural projects, and they're transparent to managers who don't want them cluttered with non-user facing stories.
I've run into managers who didn't want any TODOs in code, they wanted everything to go through the tracker. I've also run into managers that told me to stop putting everything into the tracker because they didn't know how to prioritize a random method refactor, and they felt like that information wasn't relevant to their scheduling.
I don't have a strong preference, but I lightly lean towards preferring the latter strategy. Issue trackers are slow. They are so slow, and so cumbersome, and so hard to organize, and you waste so much time linking to code files that get refactored or moved around so the context is lost. The nice thing about a TODO in code is if the method gets deleted, the TODO also gets deleted. When you're refactoring code, you don't have to go search an issue tracker and think, "wait, is there anything related to this refactor I need to update?"
If documentation is code that never gets compiled, issue trackers are like dynamically linked libraries that never actually get compiled or linked. It's very hard to keep them up-to-date; it's very hard to preserve the references.
For smaller, single-person independent projects, I don't use issue trackers. I use high-level todo lists, and all of my notes are in code next to the context they're being used for. This is because an issue tracker is just bloat for those kinds of projects.