News for hackers. Stuff that matters.
Re:Pipenv: promises a lot, delivers very little by dfee on Wednesday December 05, 2018 @04:32PM (#18612805)
I was sorely disappointed with pipenv, and transitioned to poetry [1], with which I’ve been very satisfied. There is also some commentary in the README on the design decisions re: pipenv [3]. Contrary to the author's perspective on poetry using poetry-specific sections of pyproject.toml, that's actually the proper implementation (and expected usage) coming out of PEP-518.

I also am a big fan of pyenv [3] but that’s of course to manage Python versions (not environments)

[1] https://github.com/sdispater/poetry

[2] https://github.com/sdispater/poetry#what-about-pipenv

[3] https://github.com/pyenv/pyenv

Re:Pipenv: promises a lot, delivers very little by epage on Wednesday December 05, 2018 @05:26PM (#18613306)
Poetry is one of the few things that gives me hope about the mess of python packaging.

It is also great to see the author is very responsive.

My only concern is the lack of integrated "toolchain" management (what versxon of python to use, something like rustup) that is cross platform.

Re:Pipenv: promises a lot, delivers very little by RayDonnelly on Wednesday December 05, 2018 @09:48PM (#18614943)
I agree that providing toolchains is very important.

The only non-system package manager that provides Python and its own toolchains - for Linux and macOS presently - which are used to compile every C, C++ and Fortran package, including Python itself is conda and the Anaconda Distribution.

Not doing this leads to static linking and that's inefficient and insecure.

Disclaimer: I work for Anaconda Inc.

Re:Pipenv: promises a lot, delivers very little by Nullabillity on Thursday December 06, 2018 @01:26AM (#18615872)
> The only non-system package manager that provides Python and its own toolchains - for Linux and macOS presently - which are used to compile every C, C++ and Fortran package, including Python itself is conda and the Anaconda Distribution.

Nix[0] is also perfectly usable without NixOS, and provides all of that, but has far more non-Python libraries and applications packaged. It's also not constantly trying to sell you an enterprise version...

[0]: https://nixos.org/nix/

Re:Pipenv: promises a lot, delivers very little by RayDonnelly on Thursday December 06, 2018 @04:13AM (#18616469)
Great I'll try it out, I always meant to but never got round to it. Does it work on macOS or Windows yet? What's the oldest Linux distro upon which it will run?

Not sure we constantly try to sell our Enterprise product. You could look at it less cynically as we sell an Enterprise product to allow us to provide the Anaconda Distribution for free.

Re:Pipenv: promises a lot, delivers very little by chriswarbo on Thursday December 06, 2018 @10:29AM (#18618770)
A lot of Nix users seem to use Mac, based on the stuff that comes up on the mailing list (discourse).

There's no "native" Windows support (yet), but I think it might work with some of the UNIX emulations (cygwin, mingw, wsl, etc.)

Not sure what the oldest working Linux version would be. However, NixOS has been around since 2003, so maybe quite old.

Re:Pipenv: promises a lot, delivers very little by Nullabillity on Thursday December 06, 2018 @03:04PM (#18621602)
> Does it work on macOS or Windows yet?

It runs fine on macOS. It works on WSL if you disable SQLite's write-ahead log (`echo "use-sqlite-wal = false" > /etc/nix/nix.conf` before installing), but it's much slower than running it on native Linux.

> What's the oldest Linux distro upon which it will run?

It brings its own libraries, so the primary question would be what kernel you use. I haven't verified any specific version, but you'll probably be fine. You might need to disable sandboxing though, since that makes pretty elaborate use of the various namespace systems.

Re:Pipenv: promises a lot, delivers very little by jabl on Thursday December 06, 2018 @04:54AM (#18616663)
conda is decent, but for some reason it seems to live in a weird parallel universe where people not in science have never heard about it.

While I'm generally happy with it, some gripes:

- Using it's own package format with its own repos means that for many (most) projects you can't get all dependencies from conda, but some from pypi as well.

- And it doesn't keep track of which files belong to which package. So package X will happily scribble over files installed by package Y, and vice versa, leading to either X or Y being silently broken depending on the order they were installed in! Argh! I mean, this is something dpkg/rpm/etc. figured out decades ago, it's not rocket science.

- The dependency solver seems a bit weird. Often when upgrading an environment, it will install the same version of a package with another 'build tag', then a few days later if you upgrade again, it will downgrade back to the previous build tag. Not sure if this is the fault of the dependency solver, or whether the problem is in the packages themselves.

- Similarly, there's a lot of mutual incompatibility in the repos. E.g. dependencies on openssl versions prevent upgrading, or require removal of some package etc. I think this is not so much the fault of the conda tool itself, but rather that Anaconda Inc. needs to be more picky wrt packaging policy. Again, Linux distros have been pretty good at this. E.g. https://www.debian.org/doc/debian-policy/ , https://docs.fedoraproject.org/en-US/packaging-guidelines/ .

PS: While I have above mentioned dpkg/rpm as examples to follow, it's not like those formats don't have problems either. https://nixos.org/nix/ and https://www.gnu.org/software/guix/ are perhaps the most prominent examples of 'next generation' packaging systems solving some of the problems of the old-school dpkg/rpm approaches.

Re:Pipenv: promises a lot, delivers very little by O5vYtytb on Wednesday December 05, 2018 @09:35PM (#18614899)
sdispater has an open issue for controlling the python version - https://github.com/sdispater/poetry/issues/621 - should make it into 1.0?
Re:Pipenv: promises a lot, delivers very little by beagle3 on Thursday December 06, 2018 @04:47AM (#18616633)
As you have experience with pyenv, pipenv and poetry, you might be able to answer: Why not conda?

I've been using conda since ~4 years now, and every single complaint lodged against any of the other package managers was never an issue with conda in the first place. And yet, it seems like there's a SEP field around it and people just ignore its existence?

In this thread, for the first time, I've seen someone mention that you might have problem porting a conda env from a Mac to Linux - never had that problem myself, but I guess it's possible; But that's easily solvable, and certainly does not require a new package manager?

Re:Pipenv: promises a lot, delivers very little by stochastic_monk on Thursday December 06, 2018 @08:16AM (#18617723)
That's about as long as I've been using conda. But I avoid it whenever I can because of a variety of gripes. The two biggest are the following:

1. On Lustre file systems, 'Solving environment...' can take minutes. I don't like waiting minutes to provide permission to install packages.

2. A lot of conda packages are broken. It's managed by maintainers, and unfortunately, some people maintaining believe that if it works on their machine, it will work elsewhere. Since I'm tired of ABI and runtime linking errors, I often just install from source.

Re:Pipenv: promises a lot, delivers very little by beagle3 on Thursday December 06, 2018 @05:38PM (#18623142)
conda doesn't require you to use conda packages; pip installs are perfectly integrated.

w.r.t lustre - can't comment about that; I prefer local file systems for development for various reasons (most importantly: mmaping huge local files is 10x to 100x more efficient than through networked filing systems).

Re:Pipenv: promises a lot, delivers very little by azag0 on Thursday December 06, 2018 @06:39AM (#18617137)
Because publishing a package on PyPI is trivial compared to publishing on the Anaconda cloud. For the latter, you need to build your package for every platform and Python version you want to support. So sou need to setup some CI. This is just such an overkill for a pure-Python package. It makes a lot of sense for a project with a nontrivial C extension though, and I do build a conda package for one such project of mine. For everything else, I publish just on PyPI.
Re:Pipenv: promises a lot, delivers very little by beagle3 on Thursday December 06, 2018 @07:24AM (#18617378)
Are you aware that Conda fully supports and integrates pip?

Publish on PyPI, it's just as usable in Conda to everyone.

But assuming you for some reason insist on publishing to the system you use - the vast majority of users don't ever publish a package; what's stopping them from using Conda?

I admit I have never tried to publish anything on the Anaconda cloud, but I'm a bit surprised - I was under the impression that publishing pure python packages is simple; The requirement to do it for different python versions, though, seems perfectly warranted to me - and indeed, I ran into issues with packages on PyPI not working on specific versions (but nowhere listed as such).

Re:Pipenv: promises a lot, delivers very little by RayDonnelly on Thursday December 06, 2018 @01:21PM (#18620601)
There's no such requirement at all, but it's a noble goal and one every package developer should strive for.
Re:Pipenv: promises a lot, delivers very little by orf on Thursday December 06, 2018 @07:38AM (#18617479)
Conda has no lockfile. The management commands are horrible to use and inconsistent. The "pip integration" is just "produce a requirements.txt and install it", so unless you're using some conda only dependency (thankfully those are few and far between) what's the point?

Conda is also annoyingly outside the python ecosystem. It seems to want to replace pip/pypi instead of working with it.

Re:Pipenv: promises a lot, delivers very little by lvh on Wednesday December 05, 2018 @05:02PM (#18613097)
The pyenv-virtualenv extension makes all of the pyenv magic work for environments (as in virtualenvs) too.
Re:Pipenv: promises a lot, delivers very little by djrobstep on Wednesday December 05, 2018 @04:48PM (#18612949)
I ran into a lot of bugs with poetry when I first moved to it (mostly with git dependencies) but they seem to have been fixed promptly, except that dependency resolution with git dependencies is still super slow.

An important thing for me is that you can disable the virtualenv management and manage it yourself (poetry makes it hard to install different python versions so I'd rather do that myself). Overall it works really well and I'd highly recommend it for general python dev.

Re:Pipenv: promises a lot, delivers very little by pytester on Wednesday December 05, 2018 @05:09PM (#18613146)
>I also am a big fan of pyenv

I like the idea of pyenv, but in practice I find it to be pretty buggy (e.g. failing when installing older versions of python due to openssl build issues). I kind of wish it had some competition.

Re:Pipenv: promises a lot, delivers very little by sweatersong on Wednesday December 05, 2018 @05:01PM (#18613075)
I'm looking into transitioning to poetry from Pipenv...my main gripes with Pipenv are 1) pyenv integration is cludgey and 2) "hard" conflicts between packages are unresolveable in pipenv and means you cannot lock your environment.

As far as #2 is concerned, does poetry allow you "ignore" or change subdependency requirements for specific packages a la maven?

Re:Pipenv: promises a lot, delivers very little by openbasic on Wednesday December 05, 2018 @05:04PM (#18613110)
Does poetry create a local vendor folder like "node_modules" or "vendor"?
Re:Pipenv: promises a lot, delivers very little by ergo14 on Thursday December 06, 2018 @02:56AM (#18616210)
That is the role of virtual env if I understand correctly what you are asking about.
Re:Pipenv: promises a lot, delivers very little by pmoriarty on Wednesday December 05, 2018 @04:56PM (#18613032)
"I was sorely disappointed with pipenv"

Can you flesh this out a little bit? What in particular were you disappointed with pipenv?

How does poetry do a better job?

Re:Pipenv: promises a lot, delivers very little by vqrs on Wednesday December 12, 2018 @05:38AM (#18662758)
I just tried poetry because of this thread. So I can't really talk about poetry, but I can share my frustration with pipenv.

First, the best thing I like about pipenv is `pipenv shell`. It's integration with virtualenvs is really good and a joy to use.

I think the CLI usage is really confusing. Every time I wanted to do something, I googled what the correct way was and often found github issues about stuff I was struggling with.

I don't think the defaults are intuitive, it's not clear when pipenv actually touches the lock file, when it just reinstalls everything and likes to make me wait 10 minutes. I don't understand why `pipenv install something` will also often also touch unrelated packages, why `pipenv install` even changes the lock file. Yes, I learned that I'm supposed to use `pipenv sync`, but whose idea was that? It's not even mentioned in the basic concepts, so maybe I still misunderstood something?

When I found poetry because of this thread, it was like someone had read my mind: https://github.com/sdispater/poetry#what-about-pipenv

I had hope when I found this: https://github.com/pypa/pipenv/issues/1463#issuecomment-3677... but my use of pipenv this past month was still very frustrating. Also, I don't really see how I can use pipenv on windows and on linux for the same project when the OS is stored inside the lock file.

Re:Pipenv: promises a lot, delivers very little by sametmax on Wednesday December 05, 2018 @05:19PM (#18613248)
Peotry does not handle the virtualenv at all. It deals only with the dependancy file and packaging.

So you usually couple it without something else. I use "pew".

I wish a tool would merge both.

Re:Pipenv: promises a lot, delivers very little by sdispater on Wednesday December 05, 2018 @05:22PM (#18613283)
What do you mean?

Poetry does create virtualenvs to install dependencies on a per-project basis.

Re:Pipenv: promises a lot, delivers very little by sametmax on Wednesday December 05, 2018 @05:27PM (#18613321)
Sorry, I got confused with another tool.
Re:Pipenv: promises a lot, delivers very little by carapace on Wednesday December 05, 2018 @07:56PM (#18614413)
> This is the recommended way of installing poetry.

    curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python


Nope.