Use Gnu Stow to keep the random bits contained in their own app directory that is symlinked into the /usr/local tree. Then you can manage everything without leaving orphan files behind.