It's problematic because plenty of people host websites and other network services on their own infrastructure at home / work, all in the residential IP space.