News for hackers. Stuff that matters.
Re:Python Environment by scrollaway on Monday April 30, 2018 @11:16AM (#16959278)
Man it's frustrating to see this comic and read the comments here, because it's all so true. Python, in its noble quest for backwards compatibility, has accumulated so many different ways of packaging, distributing and installing libraries and apps that it's rivaling Google's chat apps.

Pipenv is the most promising solution today but is still very, very new. It's modeled after yarn and has been officially blessed as "The One True Way" of installing stuff by the Python documentation. It has a way to go still to be as good as yarn (especially in terms of speed). The Python ecosystem has never had proper declarative packages like package.json (setup.cfg can be used to have fully declarative package metadata, but I seem to be the only one using it that way), which is a problem for package managers.

To those suggesting it, Docker is great but you're still dealing with a package manager inside Docker, so that's a moot point. It avoids the need for virtualenv, kind of, but so does pipenv and it does so more reliably and reproducibly (pipenv implements lockfiles).

Re:Python Environment by wintorez on Monday April 30, 2018 @11:28AM (#16959398)
For a language that has "There should be one-- and preferably only one --obvious way to do it." as part of its core philosophy, this is very ironic.
Re:Python Environment by callahad on Monday April 30, 2018 @01:38PM (#16960583)
It's aspirational, not prescriptive.

Just consider urllib, urllib2, urllib3, and requests. Or getopt, optparse, argparse, and docopt. Or countless others.

I appreciate that the Python community continually strives for ever-more obvious and ergonomic ways of accomplishing tasks. While there is an ecosystem complexity cost to the solutions-that-weren't, it's no greater than the churn than I've seen in other popular languages.

The Zen of Python even pokes fun at the "one obvious way" by writing its three em-dashes three different ways in the same document.

Re:Python Environment by wintorez on Monday April 30, 2018 @03:31PM (#16961503)
>> The Zen of Python even pokes fun at the "one obvious way" by writing its three em-dashes three different ways in the same document.

I did not know this! Thanks :)

Re:Python Environment by spookthesunset on Monday April 30, 2018 @11:36AM (#16959475)
Was thinking the same thing. When python was getting started it was up against Perl, which toon the opposite approach and there was a myriad of ways to do something in the language.

Package managers for languages were just starting to become a thing back then. (At least I think that is true...). Now days you wouldn’t dream of releasing a new language without some kind of standard package manager...

Dunno where I’m going with this comment but it is definitely interesting to see how much has changed since then. Hell even Microsoft has a package manager for it’s languages...

Re:Python Environment by blowski on Monday April 30, 2018 @11:42AM (#16959538)
> Now days you wouldn’t dream of releasing a new language without some kind of standard package manager...

Isn't this one of the major criticisms of Go?

Re:Python Environment by Dangeranger on Monday April 30, 2018 @12:15PM (#16959827)
My feeling is that a good package manager is the exception to the rule. Particularly around managing versions of the language runtime and the versions of packages available within projects using that runtime.

Rust is the exception here in that Cargo is a very nice piece of kit.

In terms of generic runtime version managers, ASDF is very nice cross language version manager for scoping versions to projects and switching between them easily.

Re:Python Environment by curun1r on Monday April 30, 2018 @01:19PM (#16960410)
> Rust is the exception here in that Cargo is a very nice piece of kit

Also Rustup, which is pretty key to Cargo remaining flexible going forward. There's a huge advantage to having the build/dependency management piece being highly coupled with the language runtime/compiler. Making cargo work with all versions of Rust would mean having to build in backwards compability and make it much more difficult to do projects like the new modules changes.

It's a key insight that there needs to be two tools surrounding a language, not just one. You need good dependency management and good version management. Node also has this right with nvm and npm/yarn.

Re:Python Environment by coldtea on Monday April 30, 2018 @01:59PM (#16960768)
Well, it's not like Go if feature packed in other areas people ask it to either...
Re:Python Environment by HeyImAlex on Monday April 30, 2018 @11:54AM (#16959643)
Yeah, and the exact same thing is happening.
Re:Python Environment by Waterluvian on Monday April 30, 2018 @11:39AM (#16959519)
That philosophy is pure and well intentioned. It will always fight with reality.
Re:Python Environment by alexbecker on Monday April 30, 2018 @11:38AM (#16959505)
Pipenv solves a very real problem, but it has a lot of problems of its own:

- It's very slow: re-locking after updating 1 dependency often takes me ~1 minute.

- It has lots of bugs. To name a few in 11.6.9: clobbers comments in the Pipfile, inconsistently includes dependencies for other build environments in Pipfile.lock, stores the wrong index in Pipfile.lock for packages not on PyPI.

- They release multiple times per day, often breaking things in patch releases.

- Kenneth Reitz is quite unpleasant to deal with in GitHub issues, which I often have to because of the previous 2 issues.

From what I've heard, Pipenv "has been officially blessed" only insofar as its maintainer got commit access to PyPA's documentation and inserted a recommendation.

Re:Python Environment by kennethreitz on Monday April 30, 2018 @08:52PM (#16963828)
I apologize for not meeting perfect standards of mood at all times. I have bipolar disorder, for what it's worth, and working on a packaging tool is very stress-inducing.
Re:Python Environment by divbzero on Tuesday May 01, 2018 @10:35AM (#16967331)
It’s not easy to develop for large communities, and even tougher to speak frankly about oneself.

Keep up the good work.

Re:Python Environment by kennethreitz on Monday April 30, 2018 @08:53PM (#16963834)
Also:

> From what I've heard, Pipenv "has been officially blessed" only insofar as its maintainer got commit access to PyPA's documentation and inserted a recommendation.

This is blatantly false.

Re:Python Environment by jwilk on Monday April 30, 2018 @02:32PM (#16961030)
The recommendation was added by Jon Wayne Parrott:

https://github.com/pypa/python-packaging-user-guide/pull/443

He doesn't seem to be a pipenv maintainer:

https://github.com/pypa/pipenv/commits?author=jonparrott

Re:Python Environment by dvnguyen on Monday April 30, 2018 @01:10PM (#16960331)
If what you've heard is true, it could be pretty toxic. Do you have any evidences to back up?
Re:Python Environment by baldfat on Monday April 30, 2018 @11:57AM (#16959673)
Sharing your awesome Python program is its Achilles heel. I have 5 computers I work on a weekly basis. I stopped using Python and Haskell due to all of this.

Racket's exe are so simple and quick to share.

    raco exe foo.rkt

https://docs.racket-lang.org/raco/exe.html

(Edited tendon to heel)

Re:Python Environment by peatmoss on Monday April 30, 2018 @06:59PM (#16963242)
I've thought about starting a "build an ecosystem for Racket" project where I document gaps in the extended ecosystem and review best-of-breed solutions from other languages as a kind of literature review / review of prior art.

Racket has such a great story for so many things that are painful in other environments. It's an under-appreciated language.

Re:Python Environment by mark_edward on Tuesday May 01, 2018 @03:15AM (#16965129)
Would you be willing to collaborate on this? Racket was my first programming language and I miss it dearly. An excuse to write code in it would be great!
Re:Python Environment by peatmoss on Tuesday May 01, 2018 @11:30PM (#16973826)
Oh my yes. Of course. PM me, and let’s figure out the right vehicle to get started.
Re:Python Environment by baldfat on Tuesday May 01, 2018 @07:25AM (#16966121)
The only issues that this really could address. Would be nice to have a "Wall of Shame" for these gaps.

A) Racket is so enjoyable everyone builds their own solutions

B) "The view that full-fledged problem solving almost always calls for language design" Which sadly falls onto deaf ears especially people from Python.

Re:Python Environment by disconnected on Monday April 30, 2018 @12:08PM (#16959778)
Yes, packaging Python applications for distribution is yet another nightmare situation.

Back in the day, we used one of the various "freeze" applications (cxfreeze, bbfreeze, probably others) and some tools developed in-house to package and distribute them for various platforms.

It has been a while since I actually needed to do this, so I don't know what the current state of the art is.

Re:Python Environment by kbenson on Monday April 30, 2018 @12:01PM (#16959709)
> Python, in its noble quest for backwards compatibility, has accumulated so many different ways of packaging, distributing and installing libraries and apps that it's rivaling Google's chat apps.

I'm not sure it's because of backwards compatibility. Perl has amazing backwards compatibility, but doesn't seem to suffer from this.

That's not said to pump up Perl or degrade Python, but because if you misidentify the problem, your proposes solution has a much lower likelihood of fixing it.

(Then again, maybe you mean something different by "backwards compatibility" than what I thought you meant)

Re:Python Environment by collyw on Monday April 30, 2018 @01:46PM (#16960673)
As a Python dev it looks like I need to do some reading as I am still using virtualenv (it usually works fine but I have do encounter the occasional issue).

Can anyone give me a rundown of the benefits of other systems over virtualenv?

On a side note I hope Python isn't going to turn into a shambles like JavaScript - its certainly starting to look like it as far as package managers go.

Re:Python Environment by scrollaway on Tuesday May 01, 2018 @03:45AM (#16965262)
JS isn't in shambles; in fact, their package manager situation is excellent: Use yarn, period. It could be better (yarn isn't default yet), but I would love for the Python ecosystem to get to: "Use pipenv and setup.cfg, period".

But as I said, it's too young. If you use pipenv, you still have to interact with pip, so that's two binaries whose commands you have to remember. And urgh, it's so slow. And it's kinda useless for libraries... pipenv doesn't publish stuff, pipfiles aren't appropriate for packages.

But we'll get there. One day. Hopefully it won't be too late.

Re:Python Environment by collyw on Tuesday May 01, 2018 @07:30AM (#16966144)
So what are the benefits over virtualenv? You have given me the downside of another binary to manage. What do I get in return?
Re:Python Environment by scrollaway on Wednesday May 02, 2018 @02:57AM (#16974739)
I've not given you anything other than a warning that it's too young.

Pipenv is promising. That means at some point I'm fairly certain it'll be and deserve to be the standard in Python packaging. In the mean time, it doesn't.

Re:Python Environment by collyw on Wednesday May 02, 2018 @05:43AM (#16975382)
That still doesn't answer the question of what it will (eventually) give me over virtualenv.
Re:Python Environment by jwilk on Monday April 30, 2018 @11:24AM (#16959363)
> Pipenv [...] has been officially blessed as "The One True Way" of installing stuff by the Python documentation.

Oh? I don't see any mention of pipenv in the cpython git repo.

Re:Python Environment by icebraining on Monday April 30, 2018 @11:29AM (#16959405)
Re:Python Environment by yrro on Monday April 30, 2018 @12:25PM (#16959933)
Nor in the Python Packaging Authority documentation!
Re:Python Environment by boxy_brown on Monday April 30, 2018 @01:10PM (#16960325)
Pipenv is mentioned in the PYPA Python Packaging User Guide tool recommendations:

> Application dependency management

> Use Pipenv to manage library dependencies when developing Python applications. See Managing Application Dependencies for more details on using pipenv.

https://packaging.python.org/guides/tool-recommendations/

Re:Python Environment by yrro on Monday April 30, 2018 @02:33PM (#16961042)
You're quite right. I skimmed Installing Packages, which mentions it in passing is a tool for managing multiple venvs, and Packaging & Distributing Projects, which doesn't mention it at all.
Re:Python Environment by devy on Monday April 30, 2018 @11:21AM (#16959328)
> To those suggesting it, Docker is great but you're still dealing with a package manager inside Docker, so that's a moot point.

Package manager/dependency manager is must for any sort of mature mass production coding environment (e.g. cargo for Rust, Maven for Java etc.) Docker doesn't solve all that but at least you are doing it only once and replicate easily and predictably. And a lot of times you can reuse Docker images built by someone else.

Re:Python Environment by mikepurvis on Monday April 30, 2018 @02:10PM (#16960865)
If the "built by someone else" docker container contains a bunch of incomprehensible hacks or black magic, then you're really just handing yourself a nightmare down the road when it comes time to update that thing or apply security fixes.
Re:Python Environment by yrro on Monday April 30, 2018 @12:25PM (#16959920)
Please forgive me for venting my frustration here, but my reaction to this is basically: oh great, yet another blinking tool to add to the list of misery that is the Python packaging experience. May as well check it out now before I have to do it in a hurry...

First of all it's not in Debian unstable. Ok, maybe it's just super new... so I'll install it with pip. A python3 -m pip install pipenv later and let's try it out!

$ cd ~/src/nexsan-exporter/nexsan-exporter $ pipenv install Creating a virtualenv for this project… Using /usr/bin/python3.6m (3.6.5) to create virtualenv… ⠋Running virtualenv with interpreter /usr/bin/python3.6m Using base prefix '/usr' New python executable in /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6m Also creating executable in /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python Installing setuptools, pip, wheel...done.

Virtualenv location: /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG Installing dependencies from Pipfile.lock (ca72e7)… 0/0 — 00:00:00 To activate this project's virtualenv, run the following: $ pipenv shell

Cute emoji and nice colours but... ~/.local/share? For a directory that will end up containing arch-specific libraries? Ok, I guess no one gives a shit about this in the modern world, oh well. Wait... "python3.6m"? That isn't the Python interpreter I asked for... but it seems to be a hardlink to the same file as /usr/bin/python3.6 so I guess maybe this is intentional? Anyway let's check out the venv...

$ ls -l ~/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3* lrwxrwxrwx 1 yrro yrro 10 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3 -> python3.6m lrwxrwxrwx 1 yrro yrro 10 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6 -> python3.6m -rwxr-xr-x 1 yrro yrro 4576440 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6m

So this is the Python folk's what, fourteenth attempt to get this right, and they are still copying the python executable into the virtual environment instead of symlinking it in? This seems to be a regression from venv, which seemed to get this right!

Right, time to install my dependencies... according to 'pipenv' the right command for this is 'pipenv install -e .' Hm, I wonder exactly what the -e option does?

$ pipenv install --help

* no menition of -e in the output *

:sadface:

Oh well, let's just run it blind!

$ pipenv install -e . Installing -e .… ⠏ Error: An error occurred while installing -e .! Directory '.' is not installable. File 'setup.py' not found.

Maybe I screwed up and ran this from the wrong directory?

$ ls setup.py setup.py

Weird, what's going on here?

$ strace -f pipenv install -e . 2>&1 | grep setup\\.py stat("/home/yrro/src/nexsan-exporter/nexsan-exporter/setup.py", {st_mode=S_IFREG|0644, st_size=1769, ...}) = 0 stat("/home/yrro/src/nexsan-exporter/setup.py", 0x7ffc013a0200) = -1 ENOENT (No such file or directory) [pid 7365] stat("./setup.py", 0x7fff84176e30) = -1 ENOENT (No such file or directory) stat("/home/yrro/src/nexsan-exporter/setup.py", 0x7ffc013a0300) = -1 ENOENT (No such file or directory) write(2, "Directory '.' is not installable"..., 62Directory '.' is not installable. File 'setup.py' not found.

Oh FFS, I give up. I think I'll let pipenv pass me by for now.

Re:Python Environment by jdpigeon on Monday April 30, 2018 @01:44PM (#16960653)
You're doing God's work, man. Thank you for helping me avoid that headache. I'm sticking with pip
Re:Python Environment by yrro on Monday April 30, 2018 @02:34PM (#16961047)
I wouldn't go that far. I would absolutely love for some tool, and pipenv as as fine as any other, to replace everything else. But right now it doesn't seem to be ready.
Re:Python Environment by scrollaway on Tuesday May 01, 2018 @04:38AM (#16965425)
It's absolutely not ready, as I made clear in several comments :) I just said pipenv is the most promising one.

I agree it's frustrating. I'm planning to donate some of my time to the project to help the situation. I hope others can too.

Re:Python Environment by yrro on Tuesday May 01, 2018 @09:37AM (#16966925)
Well I filed these as bugs this morning and got mixed feedback; the 'install -e .' but was caused by a stale Pipefile in the parent directory that I did not create, but at least removing it makes things work. Unfortunately upstream does not consider a stale copy of the python interpreter copied to dozens of directories on the filesystem that will never be patched to be an issue, so from my PoV pipenv gets a big raspberry and I'll stick with manually running 'python -m venv' which has gotten this right since Python 3.3!
Re:Python Environment by sireat on Tuesday May 01, 2018 @05:40AM (#16965644)
What best practices do I teach to complete beginners to Python?

I am teaching adult beginners and it would be best to avoid the messy parts in the beginning.

I thought I could get by with just pip and briefly touching on virtualenv.

I sin myself by avoiding all the mess by cloning a full bare bones Lubuntu virtualization whenever I need a clean project.

Theoretically Docker should suffice but it is yet another layer of complexity.

Re:Python Environment by bmarkovic on Monday April 30, 2018 @04:17PM (#16961866)
There was Zope Buildout for years, that had declarative packaging format, used the cheese shop (biggest issue with conda) and isn't slow as molasses but it was and still is largely ignored by the larger community.
Re:Python Environment by cpburns2009 on Monday April 30, 2018 @02:37PM (#16961079)
I just started using pipenv for a small personal project, and I like it. The usage is similar to git which I prefer over the weird bash activate mangling that a normal virtualenv does.