And plus, it's just super confusing. It's never been clear to me what the difference between pip and pip3 is. I thought pip3 was for python3 packages, but it seems like pip can also be used for python3 as well. But then sometimes it seems like if I use pip I only get the python2 version.
With npm or go get, if I install a package I know exactly where it got installed a how to mnaually remove if needed. If I install something with pip, the newly installed package goes into some mysterious directory whose location is never quite certain.
So yes, there are major problems with pip, hence why competitors are showing up like poetry.
It's pretty simple; pip3 will always be for python3, pip2 will always be for python2. pip is context dependent, for example if you are in a python2 venv, it will use py2, and vice versa.
> With npm or go get, if I install a package I know exactly where it got installed a how to mnaually remove if needed. If I install something with pip, the newly installed package goes into some mysterious directory whose location is never quite certain.
If you pip install a library, it will almost definitely be installed in pythondir/lib/site-packages, unless you have messed around with your installation
Personally, I use pipenv (manages package dependencies and sub-dependencies), pyenv (python version management), and pyenv-virtualenv (creates an isolated version of Python such as 3.8) together, in tandem, over poetry and others. Personally, I think this combination gives the best amount of versatility. This even works well with difficult installs of python packages that need to be compiled into executable software. This includes stuff that has not been recently updated.
A good guide on how to do this is here (see table comparison against poetry at the bottom of the article): https://towardsdatascience.com/python-environment-101-1d68bd...
pip3 is the name of the system installation of the pip executable corresponding to the python 3.x installation on linux systems (maybe Mac, too) when installed from system package managers on systems where python3 is the name of the python 3.x executable when installed from the package manager.
This is not really a pip issue, but a “how python is packaged to support side-by-side system installations” issue.
pip install "pip<22"
To check which pip is for which python:
pip --version
Pyenv is pretty great for managing multiple python versions.
Is there something crucial I’m missing? I feel like package management has largely been solved in python, but I admittedly haven’t done a lot of research into what other workflows are out there.
Pip’s dependency solving strategy is not correct, especially if you install pip packages one at a time.
A real dependency solver would download the metadata for all matching versions (can be done with two seeks for a wheel but gotta run setup.py for an egg…) do an smt solve and then install the wheels.
Pip just starts installing packages, hopes for the best, sometimes backtracks, often gets stuck or does the wrong thing.
One is that the dependencies are a beast. The risk that it won't find a solution between a number of libraries that are only compatible with certain versions is high.
The other one is that ML projects themselves contain data, often large amounts of data.
For instance a "word2vec" style model might be 1 gigabyte and it might be a part of another model. (Say you turn the words to vectors then put the vectors through a CNN or RNN.) The Python packaging system might be a logically correct place to store this data (a necessary part of the the model) but it's a big file that will cause hassles if you do everything right, big hassles if you do things wrong (like compress anaconda packages with slow bzip2, use whatever algorithm that Docker uses to superamplify I/O, ...)
It is nice to pack all your "empty" models (ready to train) as python packages and maybe even your "trained models" (some data files to supplement the empty files) but it will take some iteration to make it all go smoothly.
I suspect my use cases haven't been as complicated as some of the others listed in this thread, which may be why I've never felt the need to look into poetry and others.
At least that's my understanding.
To gain traction and acceptance, the offering must be several times better.
Marketing also counts.
Hence the lingering demise of python2.7: python3 didn't offer sufficient sizzle until ~3.6 or so.
If there's no real problem with the existing tools, then people will not change their tested and trusted way of working for some fancy new tool, no matter what that new tool promises!
There's an old engineering proverb about that: if it works don't mess with it.