News for hackers. Stuff that matters.
Re:Python Is Eating the World by umvi on Sunday August 11, 2019 @11:13PM (#20672096)
If only its package management were as easy as its syntax...

I wish pip worked the same way as npm: -g flag installs it globally, otherwise it creates a local "python_modules" folder I can delete at any time. And optionally I can save the dependency versioning info to some package.json...

Instead, pip is a nightmarish experience where it fails half the time and I have no idea where anything is being installed to and I'm not sure if I'm supposed to use sudo or not and I'm not sure if I'm supposed to use pip or pip3, etc.

Re:Python Is Eating the World by geofft on Sunday August 11, 2019 @11:22PM (#20672134)
Here's a simple but less-than-completely-documented way to keep Python package management under control:

1. Don't install anything globally. Don't pip install --user, definitely don't sudo pip install.

2. For each project you want to work on, create a venv. Yes, there are tools for this, but the base venv tool is totally fine. (venv should be included in your Python, but a few distributors like Debian put it in a separate package - install it from them if needed.) Use python3 -m venv ~/some/directory to create a venv. From here on out

3. As a first step, upgrade pip: ~/some/directory/bin/pip install -U pip.

4. Install things with ~/some/directory/bin/pip install.

5. Run Python with ~/some/directory/bin/python.

Slightly advanced move: make a requirements.txt file (you can use .../bin/pip freeze as a starting point) and use .../pip install -r requirements.txt. That way, if you get any sort of package resolution error, you can just delete your venv and make a new one. (Downloads are cached, so this isn't super annoying to do.)

A "project" can either be actual Python development, or just a place to install some Python programs and run them out of the resulting bin/.

(Edit: Yes, I know about activate, see the replies below for why I don't recommend it. With these rules, you get to say "Never ever type pip, only .../bin/pip", which is a good safety measure.)

Re:Python Is Eating the World by nexuist on Sunday August 11, 2019 @11:29PM (#20672172)
Herein lies my problem. If I want to start a Node project I run `npm init` and then `npm install --save` to my heart's content. If I somehow manage to mess it up I just delete node_modules/ and install again.

If I want to start a Python project I have to set up venv and remember to put relative paths in front of every command or else it'll run the system version. Sounds simple, but it's still something to always remember.

Re:Python Is Eating the World by rootusrootus on Sunday August 11, 2019 @11:37PM (#20672213)
Just source the activate script and it'll prepend the correct path so that you don't need to do anything else.
Re:Python Is Eating the World by odonnellryan on Monday August 12, 2019 @05:32AM (#20673840)
You have to use something like anaconda. It's basically the same as npm_modules (sorta).
Re:Python Is Eating the World by lapnitnelav on Monday August 12, 2019 @09:50AM (#20675358)
Use virtualenvwrapper.

$ mkproject foobar

$ workon foobar

You're set

Re:Python Is Eating the World by jbmsf on Sunday August 11, 2019 @11:33PM (#20672188)
Also: set PIP_REQUIRE_VIRTUALENV=true

You won't be able to install with pip unless you are in a virtualenv.

Re:Python Is Eating the World by ausjke on Sunday August 11, 2019 @11:38PM (#20672220)

    1. pip install --user and sudo pip install are fine actually, they will not interfere with venv, they can co-exist just fine.
    2. yes
    3. probably do "source bin/activate" first, then run 'pip install -U pip'
    4. just run pip install whatever, no need the full PATH
    5. just run python directly, no need the full PATH
    6. run 'deactivate' when you're done for now, 'source bin/activate' when you want to continue/resume sometime later
in fact I like this better than node_modules, the venv layout of bin/include/lib is more natural comparing to the odd name of "node_modules" in my opinion, and I don't need npx etc to run commands under node_modules either, all are taken care by venv with its 'activate' script
Re:Python Is Eating the World by geofft on Monday August 12, 2019 @12:03AM (#20672323)
pip install --user and sudo pip install won't break your venv. But they will break your system Python and any OS commands that depend upon system Python, perhaps including pip and virtualenv themselves, which is incredibly confusing. I've helped both friends and coworkers un-break it, and the symptoms aren't generally obvious. I wrote the patch to pip in Debian to prevent sudo pip install from removing files from Debian packages via upgrading packages. It's a huge pain, it's only worth running if you know exactly what you're doing, and as someone who does know exactly what they're doing I can attest that it's never necessary. After all, you can always just make a virtualenv.

One thing I did at my last job was to make a Nagios alert for machines with files in /usr/local/lib/pythonX.Y/site-packages, indicating that someone had run a sudo pip install, which was super helpful for "why is this machine behaving slightly differently from this other machine which should be identical". We had a supported workflow involving virtualenvs and also we had multiple members of the Debian Python team on staff if you needed systemwide packages, so not only was there always a better solution, there were people to help you with that better solution. :)

Re activate/deactivate, that's a matter of taste but I find it easier to avoid it completely too - see my reply in https://news.ycombinator.com/item?id=20672299 for why. Basically, you get the simple rule of "Never run bare pip" instead of "Remember which pip is your current pip and whether it's the one you meant."

Re:Python Is Eating the World by Smithalicious on Monday August 12, 2019 @12:13AM (#20672369)
> But they will break your system Python and any OS commands that depend upon system Python

Sudo pip install might on some distros (and I consider this to be a bug on the distro level, not a Python issue) but I've never heard of --user breaking anything

Re:Python Is Eating the World by geofft on Monday August 12, 2019 @12:55AM (#20672539)
Maybe I'm misremembering, but, isn't the point of pip install --user to get things onto your import path when running the base Python interpreter, just like sudo pip install would (except scoped to your user)? If so, wouldn't installing an incompatible newer version of some library (or worse, a broken version) break system commands that import that library, when that same user is running those commands?
Re:Python Is Eating the World by scriptdevil on Sunday August 11, 2019 @11:29PM (#20672173)
$ source some/directory/bin/activate

And you don't need to keep typing out the full directory name. When done with the environment

$ deactivate

Re:Python Is Eating the World by geofft on Sunday August 11, 2019 @11:58PM (#20672299)
I've grown to dislike activate, because it breaks the simple rule of "never run pip, python, etc., only run your-venv/bin/pip, python, etc.". Now the rule is "Don't run pip, python, etc., unless you've previously run activate and not deactivate" - and it has the complicated special case of "make sure the command exists in your virtualenv." (For instance, it's definitely possible to have a Python 2 virtualenv where pip exists but not pip3, and now if you run pip3 install from "inside" your virtualenv it's global pip! Or you might have a Python 3.6 virtualenv and type python3.7 and wonder where your packages went, or several other scenarios.)

If you have the shell prompt integration to remind you whether you're in a virtualenv or not, it's fine, but I don't always have it, and I find it helpful to manually type out the full directory name (generally with the help of up-arrow or tab...) so I know exactly what I'm running.

Re:Python Is Eating the World by ausjke on Monday August 12, 2019 @12:13AM (#20672364)
for bash/fish it automatically prefix with a (your-venv-name) so it's obvious you're under some venv, not sure about csh but I would assume it will do something similar. looks like venv supports bash/csh/fish only by default however.
Re:Python Is Eating the World by WoodenChair on Sunday August 11, 2019 @11:54PM (#20672284)
This is good advice, but it elucidates the problem. There should be 2 steps (1. list your dependencies, 2. pip install). Not 5.
Re:Python Is Eating the World by geofft on Monday August 12, 2019 @12:20AM (#20672391)
I agree. I'm optimistic about tools like Poetry https://poetry.eustace.io/docs/basic-usage/ for solving this. Unfortunately Python predates the realization that good packaging tools were a thing that needs to be solved in the core language and not externally (Go, Rust, Node, etc. postdate this realization; C, Perl, Java, etc. also predate it).

The flip side is that decoupling the interpreter/compiler from the build system makes it more possible to write tools like Poetry (and, indeed, virtualenv) that explore new approaches to the problem. At my day job where we do hermetic in-house builds with no internet access and (ideally) everything built from checked-in sources, building C and Java and Python is straightforward, because we can just tell them to use our source tree for dependency discovery and nothing else, and we can set up CFLAGS / CLASSPATH / PYTHONPATH / etc. as needed. Building Go and Rust and Node is much more challenging, because idiomatic use of those languages requires using their build tools, which often want to do things like copy all their dependencies to a subdirectory or grab resources from the internet.

Of course, given that it's Python, there should be one - and preferably only one - obvious way to do it....

Re:Python Is Eating the World by Groxx on Monday August 12, 2019 @01:26AM (#20672671)
>Unfortunately Python predates the realization that good packaging tools were a thing that needs to be solved in the core language and not externally (Go, Rust, Node, etc. postdate this realization; C, Perl, Java, etc. also predate it).

Sure, but it's also a cultural thing. Ruby is nearly as old, and also predates this, but has nowhere near the insanity of Python. The community jumped on bundler and rvm/rbenv/etc super quickly, and rapidly improved them, while the Python community is barely even aware of pip-tools / pipenv AFAICT. Even virtualenv is really only a "real pythoners know" thing, it's rarely mentioned in guides, so newbies screw up their global environment frequently.

Re:Python Is Eating the World by TomK32 on Monday August 12, 2019 @02:39AM (#20672994)
Ruby was exotic until someone translated the docs to English, but the whole ecosystem is indeed one reason I love Ruby. I really don't understand why python 2.7 is still a thing 11 years later. Sure, legacy systems, but if I install recently active open source on my machine I wouldn't expect it to use an outdated version of a programming language. Upgrading can't be that hard.
Re:Python Is Eating the World by afarrell on Monday August 12, 2019 @03:52AM (#20673386)
python 2.7 is still a thing because it was announced as the LTS* release over a decade ago.

* nobody called it that, but thats effectively what it meant to say “there will be no python 2.8. Python 2.7 will be supported until T. Python 3.x will be superceded in a year.”

Re:Python Is Eating the World by sametmax on Sunday August 11, 2019 @11:22PM (#20672128)
Use virtualenv. Always.

Python 3 comes with it as "python -m venv". Once in the virtualenv, you don't have to worry about the various pip forms and effects, you can just pio install.

You can get fancier than that of course, but that's what works with stock python, on all OS.

Re:Python Is Eating the World by vonseel on Sunday August 11, 2019 @11:34PM (#20672203)
Meanwhile, all the python developers who have started doing any Js work are saying “I wish npm worked as easy as pip/virtualenv”...

It really isn’t that difficult, it’s just different. Different always seems wrong, at first.

Re:Python Is Eating the World by odonnellryan on Monday August 12, 2019 @05:34AM (#20673852)
I haven't seriously JavaScript'd in a couple of years but my problem with it then was different versions of node or npm. Nice thing about python virtual environments is that problem never exists (can make environments with whatever version you want).
Re:Python Is Eating the World by vonseel on Monday August 12, 2019 @11:04AM (#20676042)
pipenv really solves the Python version problem, IIRC. I don't actually use pipenv, myself, since I haven't had time to thoroughly figure out the new magic and I prefer to know exactly what's going on with my Python's.

npm doesn't solve the duplicating of deps any better than python/pip, as far as I know. The react-native app I'm currently working on has a node_modules folder at 960Mb. That's probably bigger than nearly every virtualenv I've ever seen. A react-native node_modules on a bare project with `--template typescript` is at least 350Mb (created one a few minutes ago). I'm using nvm for node version management. No problems so far.

Re:Python Is Eating the World by _hardwaregeek on Sunday August 11, 2019 @11:34PM (#20672197)
Exactly. NPM gets a lot of hate but lockfiles and local install by default is great. The default mode should not be global installation. Also imo virtual environments aren't amazing. Having some mode you have to remember to flip on that changes essentially the semantics of your pip commands seems a little brittle. Tools that work on top of pip and venv like Pipenv or Poetry seem a lot better.
Re:Python Is Eating the World by yawaramin on Monday August 12, 2019 @12:57AM (#20672546)
The default should actually be a shared cache that is symlinked to local projects, like pnpm does.
Re:Python Is Eating the World by j88439h84 on Monday August 12, 2019 @12:05AM (#20672334)
Just use Poetry. https://poetry.eustace.io
Re:Python Is Eating the World by michaelmrose on Monday August 12, 2019 @12:29AM (#20672422)
Man that looks fantastic. It looks so useful that it ought to be an official part of Python.
Re:Python Is Eating the World by rleigh on Monday August 12, 2019 @02:11AM (#20672874)
This isn't even the start of the problems with pip and pypi. If I install pylibtiff, it embeds a copy of an old libtiff which is binary incompatible with the libraries on my system. I have to build it from source, then it works just fine. But I can't inflict this level of brokenness on my end users.

This applies to many packages embedding C libraries, including numpy, hdf5 and all the rest. There has been minimal thought put into binary compatibility here, meaning that it's a lottery if it works today, or will break in the future.

Re:Python Is Eating the World by fredthomsen on Sunday August 11, 2019 @11:24PM (#20672140)
I couldn't agree more with this. I was forced into doing some UI coding and although I could never full embrace js, the package management aspects (esp having the sane default of not installing packages globally) were definitely superior to python.
Re:Python Is Eating the World by kingbirdy on Sunday August 11, 2019 @11:20PM (#20672118)
That's what virtualenvs are for. Admittedly they're not a perfect solution but the tooling is quite good nowadays.
Re:Python Is Eating the World by jbay808 on Sunday August 11, 2019 @11:18PM (#20672110)
Use Conda envs!
Re:Python Is Eating the World by tachyonbeam on Sunday August 11, 2019 @11:20PM (#20672123)
I feel uncomfortable with the fact that people feel a third-party solution is the best way to solve this mess. It can also get messy when packages installed with pip, pip3, conda and apt are all entangled with one another in various ways.
Re:Python Is Eating the World by adamson on Sunday August 11, 2019 @11:25PM (#20672148)
It’s unfortunate that it’s third party, but conda has the unquestionable advantage of being the only Python-centric packaging system that has a reasonable shared binary library story
Re:Python Is Eating the World by geofft on Monday August 12, 2019 @12:27AM (#20672417)
I'm curious, do you not find wheels + manylinux reasonable? I agree that until recently, Conda definitely had that advantage, but now that you can `pip install scipy` and have that get you a working library and not try to compile things on your machine what does Conda offer beyond that?

I guess one thing Conda has that the pip ecosystem doesn't is that it supports installing non-Python shared libraries like libcurl on their own. Is that an advantage? (We absolutely could replicate that in the pip ecosystem if that was worth doing, and it's even not totally unprecedented to have non-Python binaries on PyPI.)

Re:Python Is Eating the World by tachyonbeam on Monday August 12, 2019 @12:15PM (#20676809)
I think it would definitely be great if pip could install non-python dependencies. One problem right now is that many projects will tell you to just pip install xyz. You execute that, things start building, and the process fails partway with some cryptic message because you're missing an external dependency. You figure out which one, you install it, start again, and another dependency is missing. Rinse and repeat. It's definitely not a turnkey solution, and this issue trips up newcomers all the time.

With respect to versioning, I think pip should be way more strict. It should force you to freeze dependency versions before uploading to pipy, not accept "libxyz > 3.5", but require a fixed range or single version. That would make packages much less likely to break later because newer versions of their dependencies don't work the same way anymore.

Re:Python Is Eating the World by jbay808 on Monday August 12, 2019 @12:59AM (#20672559)
Does pip allow version number dependencies? Conda is able to upgrade/downgrade packages to resolve conflicts, whereas pip just seems to check if a package exists and shrugs when there's a version conflict.
Re:Python Is Eating the World by geofft on Monday August 12, 2019 @01:15AM (#20672624)
pip does handle versioned dependencies and ranges, and know enough to upgrade existing packages when needed to resolve an upgrade. Its resolver isn't currently as complete as Conda's - see https://github.com/pypa/pip/issues/988 . (On the other hand, the fact that Conda uses a real constraint solver has apparently been causing trouble at my day job when it gets stuck exploring some area of the solution space and doesn't install your packages.... so for both pip and conda you're probably better off not relying too hard on dependency resolution and specifying the versions of as many things as you can.)
Re:Python Is Eating the World by krapht on Sunday August 11, 2019 @11:26PM (#20672151)
... the same thing happens if you mix stuff in your usr/bin directory that isn't managed by your system package manager.

The solution is: don't mix your package environments. Use a conda environment. Just like in Linux, you'd use a container. If you wait for the Python steering committee to fix pip you'll be waiting a long time.

Re:Python Is Eating the World by sidlls on Monday August 12, 2019 @12:04AM (#20672327)
Conda doesn't really solve the packaging problems of python. It can make them worse if conda doesn't have a package you need.
Re:Python Is Eating the World by dagw on Monday August 12, 2019 @03:27AM (#20673250)
Conda supports pip installing into a conda environment if there is no conda package.
Re:Python Is Eating the World by sidlls on Monday August 12, 2019 @11:29AM (#20676310)
Yes, exactly my point: then you have to potentially deal with conflicting dependencies between pip and conda packages. This happens and it's a pain to deal with.
Re:Python Is Eating the World by KaiserPro on Monday August 12, 2019 @03:24AM (#20673230)
please no.

As someone who has to look after a bunch of servers used by researchers, conda is the biggest cause of things failing for everyone.

conda installs stuff all over the place and its very easy to install conflicting stuff in the same place.

I just wish that either venv was automatic, or at least the second thing you learn in python.

Re:Python Is Eating the World by jbay808 on Monday August 12, 2019 @07:07PM (#20681070)
Are you sure you're talking about Conda's virtual envs? Everything gets installed into the folder /envs/my-env-name.
Re:Python Is Eating the World by gedy on Sunday August 11, 2019 @11:23PM (#20672138)
I agree but is it intentional? Seems like a lot of folks rail about how npm works (not just because of the "micro package" philosophy)
Re:Python Is Eating the World by dangom on Sunday August 11, 2019 @11:19PM (#20672113)
Is there any particular bottleneck towards getting this to work with Python? Would not a simple pip wrapper be enough for the task?
Re:Python Is Eating the World by cfors on Sunday August 11, 2019 @11:23PM (#20672136)
What you are describing is a virtualenv and a requirements.txt file. There are tons of other options for Python that do the same thing.
Re:Python Is Eating the World by gilad on Sunday August 11, 2019 @11:19PM (#20672117)
Packaging is definitely recognized as an issue; e.g. here: https://cecinestpasun.com/entries/where-do-you-see-python-in...