Yes, the packages that pip installs contain a setup.py (created by the package author) and pip will run that as you when you're doing your pip install. The setup.py could do arbitrary bad things to you, like leaking your ssh keys, deleting your files, or whatever.