Whenever I install directly instead of using e.g. apt (typically because I want a more current version than the distro has), I use GNU stow [0], for precisely this reason. It works at the system level, at the user level, within projects, wherever. A simple "stow -D" is all it takes to clean out all the directories.

[0] https://www.gnu.org/software/stow/