I give a lot of Python professional trainings, and I get to do that regularly, in very diverse situations. It's indeed full of gotchas.
Since it's not going to be solved quickly, meanwhile, here is what works if you need to help people setuping python:
1 - Install Python correctly
The first version Python download link for windows is 32 bits. You want 64 bits, so you should actually not click on that. Also, make sure they use the latest minor release possible, as early ones can have weird bugs. Tell people to install from the app store if they are on windows 10, or give them a link to the proper (non web) installer.
Linux: python versions may not be available for their linux distro. Use EPEL for Centos or deadsnake for Ubunto. Other Linux users chose something exotic and must be able to deal with it.
Mac: brew is fine. Official installer too.
Cause: Python support is crazy good. It support 32 bits. 3.4 supports Windows XP. 2.7 supported Atari and Solaris! So there are a lot of installers, and a lot of versions. But also because the official Python website does a poor job at directing the user.
2 - Run Python correctly
This is the great lie of Python running. You cannot just use the "python" command, which is what every doc and tutorials tell you to do.
On linux and mac, tell them to use the suffixed PythonX.Y command, with X.Y being the version of python they need. E.G: python3.6.
On windows, tell them to use the "py -X.Y" command. E.G: py -3.6
Tell them anytime they see a tutorial with "python" in it, they should replace it mentally with "PythonX.Y" or "py -X.Y" depending of who you have in front of you.
Cause: people often ends up with have several versions of Python installed on the same machine, so you can't tell them to just use the "python" command. Of course, windows and unix never agreed an on naming convention. What's more, the Windows situation can lead to a PATH problem, and "python" may not be found, while "py" will always be. We are talking about providing the "py" command everywhere.
3 - Install tools correctly
Introduce them to pip. Tell them to never, ever install stuff using admin rights with it, even if told to by documentation or tutorials. No "sudo". No "run console as admin".
If you need to install a tool, such as black, mypy, pylint, etc., outside of a venv, use "--user" to install it for the current user. This requires no admin rights.
Also, don't use the pip command directly. Use "-m" so that you always know from which python you are installing the command for. E.G:
python3.6 -m pip install black --user # unix
py -3.6 -m pip install black --user # windows
Cause: installing with admin rights can destroy your python installation. Also, pip may not be in the PATH, or be attached to the wrong version of Python. It's also because we are twisting a lib packaging tool into providing programs.
This is not specific to python. Node had to introduce a whole know command, npx, to solve the same problem, and the ones bellow. It actually takes a command name, and if it doesn't exist locally, but is registered on npm, download the package that seems to contain it, install it in a temp folder with all its deps, then immediately attempt to run it in isolation.
4 - Use tools correctly
It's another lie from docs and tutorials. If you install a tool, just calling the command may fail.
If you installed a command outside of a venv like above, then you should call it using "-m". E.G:
python3.6 -m black # or py -3.6 -m black
Cause: again, you don't know how the PATH is setup, so there is no guaranty that the command will be available, or ran from the proper python version. People can have completely messed up machines and there is nothing you can do about it. So don't depend on the PATH.
5 - Use venv correctly
The solution to all those shenanigans are the venv. Once you are in a venv, you don't need to tell the version of python, you don't need "py", "-m" or "--user". You can call commands directly. You can just use "python".
But first, people need venv installed. It's installed by default on Windows and Mac, but on linux, it's often a package to install like python3-venv, python3-pip or python3-setuptool.
Do not "pip install virtualenv": people will get confused between the tools.
So, make them create a venv: "pythonX.Y -m venv name_of_the_venv" (or py -X.Y).
Show them how to use the python/pip from the venv WITHOUT activating it, so that they understand how it works and show them the that unix have ./bin and windows ./Scripts.
Then show activation, and tell them to install pytest/jupyter and Co inside. Show them that it solves all the previous problems, so that they are motivated to use them.
Tell them to NOT put their code in the venv folder. And that they can't move or rename a venv. Show them "pip freeze > requirements.txt" + "pip install -r requirements.txt" as an alternative.
Cause: historically python didn't have the venv module, and people used to pip install virtualenv, which mean we still have docs about it, and the name stuck. Having a long history makes things complicated. And once again, the differences between Windows and Unix bites us, but that's the cost of being portable. It's not Python-specific. The fact you can't move a venv is an artefact of the venv design that the community never solved.
# Variant
People may want or need to use anaconda or the python embedded in a system (blender, qgis, etc). Then it's a whole different cycle of problems and solutions, so I won't talk about it here, this post is long enought.
Again, Python diversity is playing against it: you don't have one popular python distrib like with nodejs or cRuby. You also have commercial Python distribs, and some are used a lot in the corporate world.
Diversity make things more resilient and encourage innovation. But it makes thing harder.
# Pyproject and setup.cfg
They have nothing to do with all that. They are useful if you want to _produce_ a package to share your code with other devs.
My company settled on pipenv, and we regret it. It's slow and buggy and has crazy behaviour. It's not bad enough that it's worth updating all projects (or rather, there's no good enough alternative) but I regret about once a week having advocated for it.
Note: I think you made a mistake, you say that you can rename or move a venv but then you say you can't.
Being stuck at home, I currently give training remotely. If you know people that need trained in anything Python (beginner, advanced, scripting, architecture, web, data analysis, etc.), think about me :)
> If you need to install a tool, such as black, mypy, pylint, etc., outside of a venv, use "--user" to install it for the current user. This requires no admin rights
I think you should add that on Linux distros it's better to use the package manager to install libraries, because a beginner can, and will, have conflicts with (system) packages relying on a particular version of a library. Only use pip if the package is not available, while with the like of Debian, you will have old versions of libraries, I think fast paced distros like Fedora does a good job at providing up to date python libraries with the package manager.
This is not the instructions to install Python.
Those are the instructions to teach anybody in any OS with any configuration to install python and deal with 3rd party dependencies.
E.G: if you are a Windows 10 user, instructions to install python are 2 lines:
- install from the store
- run "py -X.Z" to start python
The complication comes when you have people with Windows, and Mac, and Linux, and they already have some Python installed in various ways, and then want to install stuff with pip, etc.
Which is true for Ruby, JS, PHP, etc.
Those are the instructions to teach anybody in any OS with any configuration to install python and deal with 3rd party dependencies.
E.G: if you are a Windows 10 user, instructions are 2 lines:
- install from the store
- run "py -X.Z" to start python
That's it.
The difficulty is to have general instructions for all cases, including packaging. Which is a problem with every scripting languages.
I too regularly have to do the same thing, but I have absolutely no issues getting newcomers onto the python ecosystem. You download the installer, you run it, and it works. Or you apt/yum install it with no issues.
Setting up a dev environment for them though is where some hassle comes. VSCode being a particularly bad one in that regard. But with Pycharm, newcomers are on boarded in a matter of minutes.
The hidden (not deleted) comments are hidden not because they contain criticism, but because of an unnecessarily harsh tone, which is unhelpful to anyone wanting to read up on the discussion at hand, and definitely not helpful for the motivation of anybody currently doing the unthankful work of maintaining pip. They do a very difficult job, where they have to deal with 20+ years of legacy, many different operating systems, many different Python distributions, many different skill levels in their user base, and even having to fight OSS fatigue due to unconstructive criticism. Pip is as good as it is due to their hard work, and we should be thankful to them.
I reread the thread last night and asked, Was I too harsh? Am I discouraging the thankless pip team? Obviously, I am biased towards myself, but no, I don't think I am the one doing the discouraging. They are the ones doing the discouraging.
Go back through the comments and look at who is PyPA official and who's not: the non-members have ideas and members shoot them down. It's not that complicated technically--there should be a switch called "require hashes" instead of an implicit requirement for hashes once you use one. But the PyPA devs just showed zero interest in understanding the problem or helping. Their first comment was that error message was fine, nothing to fix!!!
I ask myself, would I want to contribute a patch to pip? I contribute patches occasionally to OSS. Not every day, but every couple of months when something comes up that affects me. I would never bother. Why? It's obvious from their tone that the patch would be stuck in a queue and never reviewed. (This happened to a coworker at a former job. We maintained a patched version of pip for years because they wouldn't take our change.)
In contrast, I just submitted a broken patch to Go (I broke tests and said so in my patch) and the Go team took it and fixed the tests. Why did I bother to submit a broken patch to Go? Because it was pretty clear from the tone of the discussion that if I got the ball rolling, they would push it the rest of the way over the hill and I want the issue fixed in the next version of Go, so I submitted what I had even without going through and fixing all the tests (which I didn't have the expertise to do).
Now, this is not the same. Go is supported by Google and has money and developers, yada yada. But the Python Foundation is an organization. It has a payroll. It has official members. They could ask the EU for €5m grant to work with N devs for Y years fixing Python packaging… Whatever. I'm not in charge of PyPA, so it's not my problem.
But I do object to the gaslighting in implying that we, the users, are discouraging the poor, unthanked PyPA devs. No, they don't want to be helped, so they put out the vibe to make it clear we should leave them alone.
Gentoo is the only distro on which I feel comfortable using the system interpreters because the versions are slotted (you can have multiple versions) and system pip install is effectively disabled.
So on other distros I use pyenv. But trying to explain this to non-Linux geeks is hopeless.
I've taken to recommending the following: use system python/pip to install pipx (but with --user). Use that to install "global" tools. Use virtualenvwrapper to manage projects. Use pip-tools to manage frozen environments (ie. requirements.txt). I've looked into replacing the latter with pipenv (which is awful) and poetry (which is better) but I've so far been unable to generally recommend anything else.
I don't even want to think about other platforms and conda and all that.
Meanwhile, the pip project is full of developers who tell you that your problem is not a problem and you’re discouraging people by pointing out that they’re blame shifting: https://github.com/pypa/pip/issues/4995
I honestly have zero faith that things will get better without a radical change in leadership. Pyproject.toml is a good step in the right direction, but it’s been around for years. They had time for adding the walrus operator, but not for replacing the setuptools, venv, pip Cerberus.