News for hackers. Stuff that matters.
Re:Python Is Eating the World by geofft on Sunday August 11, 2019 @11:22PM (#20672134)
Here's a simple but less-than-completely-documented way to keep Python package management under control:

1. Don't install anything globally. Don't pip install --user, definitely don't sudo pip install.

2. For each project you want to work on, create a venv. Yes, there are tools for this, but the base venv tool is totally fine. (venv should be included in your Python, but a few distributors like Debian put it in a separate package - install it from them if needed.) Use python3 -m venv ~/some/directory to create a venv. From here on out

3. As a first step, upgrade pip: ~/some/directory/bin/pip install -U pip.

4. Install things with ~/some/directory/bin/pip install.

5. Run Python with ~/some/directory/bin/python.

Slightly advanced move: make a requirements.txt file (you can use .../bin/pip freeze as a starting point) and use .../pip install -r requirements.txt. That way, if you get any sort of package resolution error, you can just delete your venv and make a new one. (Downloads are cached, so this isn't super annoying to do.)

A "project" can either be actual Python development, or just a place to install some Python programs and run them out of the resulting bin/.

(Edit: Yes, I know about activate, see the replies below for why I don't recommend it. With these rules, you get to say "Never ever type pip, only .../bin/pip", which is a good safety measure.)

Re:Python Is Eating the World by nexuist on Sunday August 11, 2019 @11:29PM (#20672172)
Herein lies my problem. If I want to start a Node project I run `npm init` and then `npm install --save` to my heart's content. If I somehow manage to mess it up I just delete node_modules/ and install again.

If I want to start a Python project I have to set up venv and remember to put relative paths in front of every command or else it'll run the system version. Sounds simple, but it's still something to always remember.

Re:Python Is Eating the World by rootusrootus on Sunday August 11, 2019 @11:37PM (#20672213)
Just source the activate script and it'll prepend the correct path so that you don't need to do anything else.
Re:Python Is Eating the World by odonnellryan on Monday August 12, 2019 @05:32AM (#20673840)
You have to use something like anaconda. It's basically the same as npm_modules (sorta).
Re:Python Is Eating the World by lapnitnelav on Monday August 12, 2019 @09:50AM (#20675358)
Use virtualenvwrapper.

$ mkproject foobar

$ workon foobar

You're set

Re:Python Is Eating the World by jbmsf on Sunday August 11, 2019 @11:33PM (#20672188)
Also: set PIP_REQUIRE_VIRTUALENV=true

You won't be able to install with pip unless you are in a virtualenv.

Re:Python Is Eating the World by ausjke on Sunday August 11, 2019 @11:38PM (#20672220)

    1. pip install --user and sudo pip install are fine actually, they will not interfere with venv, they can co-exist just fine.
    2. yes
    3. probably do "source bin/activate" first, then run 'pip install -U pip'
    4. just run pip install whatever, no need the full PATH
    5. just run python directly, no need the full PATH
    6. run 'deactivate' when you're done for now, 'source bin/activate' when you want to continue/resume sometime later
in fact I like this better than node_modules, the venv layout of bin/include/lib is more natural comparing to the odd name of "node_modules" in my opinion, and I don't need npx etc to run commands under node_modules either, all are taken care by venv with its 'activate' script
Re:Python Is Eating the World by geofft on Monday August 12, 2019 @12:03AM (#20672323)
pip install --user and sudo pip install won't break your venv. But they will break your system Python and any OS commands that depend upon system Python, perhaps including pip and virtualenv themselves, which is incredibly confusing. I've helped both friends and coworkers un-break it, and the symptoms aren't generally obvious. I wrote the patch to pip in Debian to prevent sudo pip install from removing files from Debian packages via upgrading packages. It's a huge pain, it's only worth running if you know exactly what you're doing, and as someone who does know exactly what they're doing I can attest that it's never necessary. After all, you can always just make a virtualenv.

One thing I did at my last job was to make a Nagios alert for machines with files in /usr/local/lib/pythonX.Y/site-packages, indicating that someone had run a sudo pip install, which was super helpful for "why is this machine behaving slightly differently from this other machine which should be identical". We had a supported workflow involving virtualenvs and also we had multiple members of the Debian Python team on staff if you needed systemwide packages, so not only was there always a better solution, there were people to help you with that better solution. :)

Re activate/deactivate, that's a matter of taste but I find it easier to avoid it completely too - see my reply in https://news.ycombinator.com/item?id=20672299 for why. Basically, you get the simple rule of "Never run bare pip" instead of "Remember which pip is your current pip and whether it's the one you meant."

Re:Python Is Eating the World by Smithalicious on Monday August 12, 2019 @12:13AM (#20672369)
> But they will break your system Python and any OS commands that depend upon system Python

Sudo pip install might on some distros (and I consider this to be a bug on the distro level, not a Python issue) but I've never heard of --user breaking anything

Re:Python Is Eating the World by geofft on Monday August 12, 2019 @12:55AM (#20672539)
Maybe I'm misremembering, but, isn't the point of pip install --user to get things onto your import path when running the base Python interpreter, just like sudo pip install would (except scoped to your user)? If so, wouldn't installing an incompatible newer version of some library (or worse, a broken version) break system commands that import that library, when that same user is running those commands?
Re:Python Is Eating the World by scriptdevil on Sunday August 11, 2019 @11:29PM (#20672173)
$ source some/directory/bin/activate

And you don't need to keep typing out the full directory name. When done with the environment

$ deactivate

Re:Python Is Eating the World by geofft on Sunday August 11, 2019 @11:58PM (#20672299)
I've grown to dislike activate, because it breaks the simple rule of "never run pip, python, etc., only run your-venv/bin/pip, python, etc.". Now the rule is "Don't run pip, python, etc., unless you've previously run activate and not deactivate" - and it has the complicated special case of "make sure the command exists in your virtualenv." (For instance, it's definitely possible to have a Python 2 virtualenv where pip exists but not pip3, and now if you run pip3 install from "inside" your virtualenv it's global pip! Or you might have a Python 3.6 virtualenv and type python3.7 and wonder where your packages went, or several other scenarios.)

If you have the shell prompt integration to remind you whether you're in a virtualenv or not, it's fine, but I don't always have it, and I find it helpful to manually type out the full directory name (generally with the help of up-arrow or tab...) so I know exactly what I'm running.

Re:Python Is Eating the World by ausjke on Monday August 12, 2019 @12:13AM (#20672364)
for bash/fish it automatically prefix with a (your-venv-name) so it's obvious you're under some venv, not sure about csh but I would assume it will do something similar. looks like venv supports bash/csh/fish only by default however.
Re:Python Is Eating the World by WoodenChair on Sunday August 11, 2019 @11:54PM (#20672284)
This is good advice, but it elucidates the problem. There should be 2 steps (1. list your dependencies, 2. pip install). Not 5.
Re:Python Is Eating the World by geofft on Monday August 12, 2019 @12:20AM (#20672391)
I agree. I'm optimistic about tools like Poetry https://poetry.eustace.io/docs/basic-usage/ for solving this. Unfortunately Python predates the realization that good packaging tools were a thing that needs to be solved in the core language and not externally (Go, Rust, Node, etc. postdate this realization; C, Perl, Java, etc. also predate it).

The flip side is that decoupling the interpreter/compiler from the build system makes it more possible to write tools like Poetry (and, indeed, virtualenv) that explore new approaches to the problem. At my day job where we do hermetic in-house builds with no internet access and (ideally) everything built from checked-in sources, building C and Java and Python is straightforward, because we can just tell them to use our source tree for dependency discovery and nothing else, and we can set up CFLAGS / CLASSPATH / PYTHONPATH / etc. as needed. Building Go and Rust and Node is much more challenging, because idiomatic use of those languages requires using their build tools, which often want to do things like copy all their dependencies to a subdirectory or grab resources from the internet.

Of course, given that it's Python, there should be one - and preferably only one - obvious way to do it....

Re:Python Is Eating the World by Groxx on Monday August 12, 2019 @01:26AM (#20672671)
>Unfortunately Python predates the realization that good packaging tools were a thing that needs to be solved in the core language and not externally (Go, Rust, Node, etc. postdate this realization; C, Perl, Java, etc. also predate it).

Sure, but it's also a cultural thing. Ruby is nearly as old, and also predates this, but has nowhere near the insanity of Python. The community jumped on bundler and rvm/rbenv/etc super quickly, and rapidly improved them, while the Python community is barely even aware of pip-tools / pipenv AFAICT. Even virtualenv is really only a "real pythoners know" thing, it's rarely mentioned in guides, so newbies screw up their global environment frequently.

Re:Python Is Eating the World by TomK32 on Monday August 12, 2019 @02:39AM (#20672994)
Ruby was exotic until someone translated the docs to English, but the whole ecosystem is indeed one reason I love Ruby. I really don't understand why python 2.7 is still a thing 11 years later. Sure, legacy systems, but if I install recently active open source on my machine I wouldn't expect it to use an outdated version of a programming language. Upgrading can't be that hard.
Re:Python Is Eating the World by afarrell on Monday August 12, 2019 @03:52AM (#20673386)
python 2.7 is still a thing because it was announced as the LTS* release over a decade ago.

* nobody called it that, but thats effectively what it meant to say “there will be no python 2.8. Python 2.7 will be supported until T. Python 3.x will be superceded in a year.”