Either you want the package manager to expect to manage everything or you don't, pick a side.
I want a system that's better at playing nice between system-managed and developer-managed packages. Distributions that stick closer to upstream (e.g. Slackware, Gentoo, or especially FreeBSD with the built-in ports system) are better than Debian for developer machines IME.
At that point you end up manually doing the job of a package manager.