1. Don't install anything globally. Don't pip install --user, definitely don't sudo pip install.
2. For each project you want to work on, create a venv. Yes, there are tools for this, but the base venv tool is totally fine. (venv should be included in your Python, but a few distributors like Debian put it in a separate package - install it from them if needed.) Use python3 -m venv ~/some/directory to create a venv. From here on out
3. As a first step, upgrade pip: ~/some/directory/bin/pip install -U pip.
4. Install things with ~/some/directory/bin/pip install.
5. Run Python with ~/some/directory/bin/python.
Slightly advanced move: make a requirements.txt file (you can use .../bin/pip freeze as a starting point) and use .../pip install -r requirements.txt. That way, if you get any sort of package resolution error, you can just delete your venv and make a new one. (Downloads are cached, so this isn't super annoying to do.)
A "project" can either be actual Python development, or just a place to install some Python programs and run them out of the resulting bin/.
(Edit: Yes, I know about activate, see the replies below for why I don't recommend it. With these rules, you get to say "Never ever type pip, only .../bin/pip", which is a good safety measure.)
If I want to start a Python project I have to set up venv and remember to put relative paths in front of every command or else it'll run the system version. Sounds simple, but it's still something to always remember.
$ mkproject foobar
$ workon foobar
You're set
You won't be able to install with pip unless you are in a virtualenv.
1. pip install --user and sudo pip install are fine actually, they will not interfere with venv, they can co-exist just fine.
2. yes
3. probably do "source bin/activate" first, then run 'pip install -U pip'
4. just run pip install whatever, no need the full PATH
5. just run python directly, no need the full PATH
6. run 'deactivate' when you're done for now, 'source bin/activate' when you want to continue/resume sometime later
in fact I like this better than node_modules, the venv layout of bin/include/lib is more natural comparing to the odd name of "node_modules" in my opinion, and I don't need npx etc to run commands under node_modules either, all are taken care by venv with its 'activate' scriptOne thing I did at my last job was to make a Nagios alert for machines with files in /usr/local/lib/pythonX.Y/site-packages, indicating that someone had run a sudo pip install, which was super helpful for "why is this machine behaving slightly differently from this other machine which should be identical". We had a supported workflow involving virtualenvs and also we had multiple members of the Debian Python team on staff if you needed systemwide packages, so not only was there always a better solution, there were people to help you with that better solution. :)
Re activate/deactivate, that's a matter of taste but I find it easier to avoid it completely too - see my reply in https://news.ycombinator.com/item?id=20672299 for why. Basically, you get the simple rule of "Never run bare pip" instead of "Remember which pip is your current pip and whether it's the one you meant."
Sudo pip install might on some distros (and I consider this to be a bug on the distro level, not a Python issue) but I've never heard of --user breaking anything
And you don't need to keep typing out the full directory name. When done with the environment
$ deactivate
If you have the shell prompt integration to remind you whether you're in a virtualenv or not, it's fine, but I don't always have it, and I find it helpful to manually type out the full directory name (generally with the help of up-arrow or tab...) so I know exactly what I'm running.
The flip side is that decoupling the interpreter/compiler from the build system makes it more possible to write tools like Poetry (and, indeed, virtualenv) that explore new approaches to the problem. At my day job where we do hermetic in-house builds with no internet access and (ideally) everything built from checked-in sources, building C and Java and Python is straightforward, because we can just tell them to use our source tree for dependency discovery and nothing else, and we can set up CFLAGS / CLASSPATH / PYTHONPATH / etc. as needed. Building Go and Rust and Node is much more challenging, because idiomatic use of those languages requires using their build tools, which often want to do things like copy all their dependencies to a subdirectory or grab resources from the internet.
Of course, given that it's Python, there should be one - and preferably only one - obvious way to do it....
Sure, but it's also a cultural thing. Ruby is nearly as old, and also predates this, but has nowhere near the insanity of Python. The community jumped on bundler and rvm/rbenv/etc super quickly, and rapidly improved them, while the Python community is barely even aware of pip-tools / pipenv AFAICT. Even virtualenv is really only a "real pythoners know" thing, it's rarely mentioned in guides, so newbies screw up their global environment frequently.
* nobody called it that, but thats effectively what it meant to say “there will be no python 2.8. Python 2.7 will be supported until T. Python 3.x will be superceded in a year.”
Python 3 comes with it as "python -m venv". Once in the virtualenv, you don't have to worry about the various pip forms and effects, you can just pio install.
You can get fancier than that of course, but that's what works with stock python, on all OS.
It really isn’t that difficult, it’s just different. Different always seems wrong, at first.
npm doesn't solve the duplicating of deps any better than python/pip, as far as I know. The react-native app I'm currently working on has a node_modules folder at 960Mb. That's probably bigger than nearly every virtualenv I've ever seen. A react-native node_modules on a bare project with `--template typescript` is at least 350Mb (created one a few minutes ago). I'm using nvm for node version management. No problems so far.
This applies to many packages embedding C libraries, including numpy, hdf5 and all the rest. There has been minimal thought put into binary compatibility here, meaning that it's a lottery if it works today, or will break in the future.
I guess one thing Conda has that the pip ecosystem doesn't is that it supports installing non-Python shared libraries like libcurl on their own. Is that an advantage? (We absolutely could replicate that in the pip ecosystem if that was worth doing, and it's even not totally unprecedented to have non-Python binaries on PyPI.)
With respect to versioning, I think pip should be way more strict. It should force you to freeze dependency versions before uploading to pipy, not accept "libxyz > 3.5", but require a fixed range or single version. That would make packages much less likely to break later because newer versions of their dependencies don't work the same way anymore.
The solution is: don't mix your package environments. Use a conda environment. Just like in Linux, you'd use a container. If you wait for the Python steering committee to fix pip you'll be waiting a long time.
As someone who has to look after a bunch of servers used by researchers, conda is the biggest cause of things failing for everyone.
conda installs stuff all over the place and its very easy to install conflicting stuff in the same place.
I just wish that either venv was automatic, or at least the second thing you learn in python.
I wish pip worked the same way as npm: -g flag installs it globally, otherwise it creates a local "python_modules" folder I can delete at any time. And optionally I can save the dependency versioning info to some package.json...
Instead, pip is a nightmarish experience where it fails half the time and I have no idea where anything is being installed to and I'm not sure if I'm supposed to use sudo or not and I'm not sure if I'm supposed to use pip or pip3, etc.