>Why? Why can't I just have the pip'ed libraries in a subdir somewhere and tell Python to look there?

Because that ties you to a single system wide python. Why would you want that?