News for hackers. Stuff that matters.
Re:Announcing Pipenv by renesd on Monday January 23, 2017 @02:00AM (#13460395)
I think python packaging has gotten LOTS better in the last few years. I find it quite pleasurable to use these days.

From binary wheels (including on different linux architectures), to things like local caching of packages (taking LOTS of load off the main servers). To the organisation github of pypa [0], to `python -m venv` working.

Also lots of work around standardising things in peps, and writing documentation for people.

I would like to applaud all the hard work people have done over the years on python packaging. It really is quite nice these days, and I look forward to all the improvements coming up (like pipenv!).

I'd suggest people checkout fades [1] (for running scripts and automatically downloading dependencies in a venv), as well as conda [2] the alternative package manager.

[0] https://github.com/pypa/

[1] https://fades.readthedocs.io/en/release-5/readme.html#what-d...

[2] http://conda.pydata.org/docs/intro.html

Re:Announcing Pipenv by sametmax on Monday January 23, 2017 @03:15AM (#13460675)
+1. Relatively to what we have before, it's so much better. But compared to the JS/Rust ecosystem, we are behind.

Now it's hard to compete with JS on some stuff : it's the only language in the most popular dev plateform (the web) and it has one implicit standardized async model by default.

It's hard to compete with rust on some stuff : it's compiled and is fast, can provide stand alone binaries easily and has a checker that can avoid many bugs.

But this. The package manager. We can compete. And yet we are late.

It's partially my fault since it's a project I had in mind for years and never took the time to work on. It's partially everybody's fault I guess :)

Re:Announcing Pipenv by Dowwie on Monday January 23, 2017 @07:14AM (#13461613)
It was definitely your fault. Thanks for nothing.
Re:Announcing Pipenv by sametmax on Monday January 23, 2017 @08:48AM (#13462111)
:) In the open source world, and given my skills, I always feel a bit responsible when a tool doesn't exist.
Re:Announcing Pipenv by nerdwaller on Tuesday January 24, 2017 @08:14PM (#13477526)
I'd strongly disagree with Python being behind relative to JS. They're roughly on par (with Python being slightly ahead due to smarter caching and more deterministic builds). Unfortunately neither does namespacing nor good versioning, especially in the cache. Take for example Maven (from Java), it caches namespaced and versioned dependencies and is pretty reliably deterministic in the build process.

Verses node which makes horribly assumptions that allow dependencies of libraries to be resolved and dumped into the same library namespace as your application. It also allows users to use the shrinkwrap feature, which imposes the dependency's settings on me (say, for example, I want to use my own npmjs proxy, this is bypassed...)

Re:Announcing Pipenv by HowDoesItWork on Monday January 23, 2017 @12:48PM (#13464082)
Wow, Javascript, really? I am guessing you don't actually work with NPM a lot.
Re:Announcing Pipenv by jessaustin on Monday January 23, 2017 @01:31PM (#13464535)
One suspects it's you who hasn't distributed or installed many modules on either python or node. So many of the problems that python has, simply don't exist for node, because it finds modules in a bottom-up hierarchical fashion. That allows a single app or module to use modules that in turn use different versions of other modules, and not to worry about what other modules are doing, or how other modules are installed, or how node is installed, or what version of node is installed. This prevents the traditional "dependency hell" that has plagued devs for decades. Thanks to tools like browserify and webpack, the browser may also benefit from this organization.

On top of all that, npm itself just does so many things right. It's quite happy to install from npm repos, from dvcs repos, from regular directories, or from anything that looks like a directory. It just needs to find a single file called "package.json". It requires no build step to prepare a module for upload to an npm repo, but it easily allows for one if that's necessary. package.json itself is basically declarative, but provides scripting hooks for imperative actions if necessary. At every opportunity, npm allows devs to do what they need to do, the easy way.

In a sense, node and npm are victims of their own quality. The types of "issues" (e.g. too many deps, too many layers of deps, too many versions of a particular dep, deps that are too trivial, etc.) about which anal code puritans complain with respect to node simply couldn't arise on other platforms, because dependency hell would cause the tower of module dependencies to collapse first. node happily chugs along, blithely ignoring the "problems".

Personally, I used to be able to build python packages for distribution, but since I've been spoiled by node and npm for several years I've found I simply can't do that for python anymore. It is so much harder.

Re:Announcing Pipenv by philsnow on Monday January 23, 2017 @02:16PM (#13465001)
npm has its own special problems. disclaimer: what I'm talking about in this post is at least six months old, which in node/npm/js world is ancient history.

> it finds modules in a bottom-up hierarchical fashion. That allows a single app or module to use modules that in turn use different versions of other modules, and not to worry about what other modules are doing

To my understanding, if your app transitively depends on package foo-1.2 in thirty different places [0], there will be thirty copies of foo-1.2 on disk under node_modules/ . Each package reads its very own copy of foo-1.2 when it require()s foo.

On a large app, that adds up to a lot of inodes ("why does it say my filesystem is full? there's only 10G of stuff on my 80G partition!" because it's used up all its inodes, not its bytes.) and a _lot_ of unnecessary I/O. The second through thirtieth copies of foo-1.2 don't come from the kernel's block cache "for free", they come from spinning rust (or if you're lucky, the dwindling number of IOps your SSD can choke out. Do you pay money for provisioned IOps?).

[0] and thirty is a lowball number for some projects, especially given the community's preference to require "leftpad" or whatever instead of writing a couple lines in their own projects

Re:Announcing Pipenv by jessaustin on Monday January 23, 2017 @03:10PM (#13465506)
...what I'm talking about in this post is at least six months old...

Haha npm@3 was out June 2015. b^)

I agree that it would have been better, on balance, for previous versions to have created hard links to already-installed modules. Actually that wouldn't be a bad option to have even now, since debugging is often easier when one has a deep directory structure to explore rather than hundreds of random names in the top-level node_modules directory. That is, if I know the problem is in module foo, I can pushd to node_modules/foo, find the problematic submodule again, and repeat until I get all the way to the bottom. [EDIT: it occurs to me that having all these hard links would make e.g. dependency version updates easier, since un-updated dependencies wouldn't have to be recopied, unix-stow-style.]

To me, the more amusing file descriptor problem is caused by the module "chokidar", which when used in naive fashion tries to set up watches on all 360 files and directories created by itself and its own 55 dependencies. At that point it's real easy to run out of file watches altogether. Some of the utilities that call chokidar do so while ignoring node_modules, but many do not.

Re:Announcing Pipenv by WorldMaker on Monday January 23, 2017 @03:35PM (#13465742)
Your concern is somewhat ancient history in the node/npm world. NPM >= 3 flattens the dependency hierarchy as much as it thinks possible. (3.0.0 was released to beta in Jun 2015; NPM >= 3 has been bundled with Node LTS since "Boron" in Oct 2016)

Subsequent versions of NPM continue to improve upon this flattening effort.

Re:Announcing Pipenv by overcast on Monday January 23, 2017 @01:18PM (#13464405)
I work with NPM everyday, and it works swimmingly. npm install packagename --save, doesn't get much easier than that.
Re:Announcing Pipenv by sjellis on Monday January 23, 2017 @07:51AM (#13461821)
This is actually one of the big problems, I think: Python packaging involves knowing a number of different things and reading various resources to get the full picture.

Recently, I built a small CLI tool in Python, and learned all of the bits needed to build, test and package my application "the right way". I knew Python syntax before, but it was a lot of effort to set this up. The difference in the experience between Python and Rust or .NET Core is actually shocking, and most it isn't down to anything that Python couldn't do, just the current state of the tooling.

Re:Announcing Pipenv by d0mine on Monday January 23, 2017 @12:35PM (#13463946)
Could you provide some specific examples of the "shocking" difference?
Re:Announcing Pipenv by sjellis on Tuesday January 24, 2017 @12:55AM (#13469132)
Python best practice: figure out the correct directory structure by reading docs and looking at GitHub repositories, learn how to write setup.py & setup.cfg & requirements.txt & MANIFEST.in files, setup py.test and tox (because Python 2 still lives), write your README in RST format (as used by nothing else ever), and for bonus points: write your own Makefile. Get depressed when you realize that target platforms either don't have Python or have the wrong version.

Rust: type "cargo new", README and doc comments in Markdown, type "cargo test" and "cargo build".

I'm being deliberately snarky, but you get the point: there has been a slow accretion of complexity over a very long time, and most of it is not the language itself.

Re:Announcing Pipenv by d0mine on Thursday January 26, 2017 @03:14PM (#13494875)
Thank you. It is exactly what I have asked for.

The Python best practice: you do not need "correct directory structure" to start programming in Python--there is no boilerplate structure to create (and that would require a tool): REPL, Jupyter notebook in a browser, a single file (module) is enough in a simple case.

There is no single "correct directory structure" for more complex cases either e.g., a Django web app and Ubuntu's Appindicator would probably have different structures and it is likely that there are tools that create corresponding project skeletons such as "django-admin startproject mysite".

Usually, the packaging/deploying are not nice & neat and differ depending on your project, chosen platform--either you follow an example or you have to study a never ending list of incomplete/contradicting standards/semi-documented rules & conventions--it is unrelated to Python (even if tools such as Ansible, Salt are implemented in Python). Naturally, there are things that can be improved.

btw, reStructuredText is a great text format for documenting Python projects (it scales from a simple README.rst to a whole book). You can always convert README.md written in your favorite Markdown flavour using pandoc or a similar tool. Markdown is great for a short human-generated plaintext content common on Github, Stack Overflow.

Re:Announcing Pipenv by sjellis on Friday January 27, 2017 @04:11AM (#13498622)
> there is no boilerplate structure to create (and that would require a tool): REPL, Jupyter notebook in a browser, a single file (module) is enough in a simple case.

Rhetorical question: is Python a scripting language or an applications language?

If it's a scripting language, then yes, a single file is all that you need. Otherwise, you need a directory structure with various support files to be able to create and build your CLI tool or library, and you need to follow conventions if you are going to publish your work as Open Source. Which in turn requires you to know which tools to use, and what they require.

Go raised the bar here, I think. Rust and .NET Core both follow the Go idea of having one top level tool that gives you what you need to test and package your library or CLI application. That means that new users can get success with at least two basic use cases very rapidly and with little effort. Which in turn means that the previous status quo is no longer so tolerable.

> btw, reStructuredText is a great text format for documenting Python projects (it scales from a simple README.rst to a whole book).

Sure, the big problem with RST is that it lost the fight. Every developer who uses GitHub has to know a little Markdown, not RST or something else. For larger documents, AsciiDoc is probably the thing. I used to be a Textile loyalist, myself, but stopped fighting against the tide and switched to Markdown some years ago.

Re:Announcing Pipenv by d0mine on Friday January 27, 2017 @09:56AM (#13500120)
Python is a general -purpose programming language. No size fits all.

You've ignored "in a simple case" and ignored the next paragraph that discussed "more complex cases." If I have a task: publish Ubuntu PPA with a package that does X, or deploy a web app on a PaaS, or add a component that massages some data for a numerical simulation on a cluster or something else. I do not see what Go, Rust, .NET do here that Python can't.

Obviously, there could be specific use-cases where some language/framework has advantages over other languages/frameworks and (as I said) there may be use-cases that can be improved that why they should be named explicitly -- that is why I've asked in the first place -- so far I don't see "shoking difference" in the general case (I don't doubt that you had experienced the shoking difference for your specific (unnamed) task).

Re:Announcing Pipenv by sjellis on Friday January 27, 2017 @06:38PM (#13503771)
My original comment was specifically about building a CLI tool, which I did as a learning exercise to practice writing a multi-file Python project that could act as a library as well as a CLI, and be packaged as a wheel for PyPI.

Please bear in mind here that I did this because I wanted to learn and follow current best practices, with a project that I specifically picked because it would not require 3rd-party libraries or big frameworks to muddle things.

I don't think that it was dissatisfying because Python is a general-purpose programming language, but because the process currently happens to involve an assortment of different tools, piecing together scattered information, and being willing to suffer a few paper cuts like Python versions and multiple old file formats. That was my experience building a simple Python 3 application, and it didn't leave me eager to do it again.

We can discuss how Go-style tooling and packaging can also help with the cases that you mentioned, but I didn't really want to stray off into an extended analysis. My original points were just that the overall developer experience that new Python users currently get compares poorly to some other languages, and that is totally fixable. I hope that things do improve, and feel that Pipenv is a very welcome move in that direction.

Re:Announcing Pipenv by d0mine on Saturday January 28, 2017 @09:00AM (#13506287)
You says "X is bad", "Y is better" but you don't say how exactly X is bad, in what way Y is better.

What is so bad about "a multi-file Python project that could act as a library as well as a CLI, and be packaged as a wheel for PyPI."

Put __init__.py in a directory and you've created a Python package where you could add as many modules including other packages as you like.

Copy-paste setup.py ("pip init" like "npm init" would help here), update it with your info and to upload a wheel to PyPI, run:

  $ pip install twine
  $ python setup.py sdist bdist_wheel
  $ twine upload dist/*
(you might need "twine register". "pip publish" would be nice).

I didn't read it but as far as I understand all of this is on one page https://packaging.python.org/distributing/

Re:Announcing Pipenv by sjellis on Sunday January 29, 2017 @06:11AM (#13513627)
> You says "X is bad", "Y is better" but you don't say how exactly X is bad, in what way Y is better.

I gave you a breakdown in my earlier comment, but I'm thinking that writing more isn't going to make this better:

* Your experience: Python does everything that you need, and quickly, because you are already very proficient in the language and the tools. * My experience: Shopping for a programming language, knows Python syntax but no attachment beyond that, has tried out a number of languages and has working knowledge of their tooling.

I'm not suggesting that Python is a bad language, just that I think that the tooling and developer experience for new users is not competitive with some other languages. If you are proficient in Python, and haven't tried other languages recently, you won't have experienced the jump: Python is slightly more hard work than Ruby or Node.js, but Rust, Go and .NET Core have a level of slickness and integration that older platforms just don't have.

> What is so bad about "a multi-file Python project that could act as a library as well as a CLI, and be packaged as a wheel for PyPI."

Again, see my previous comment for a list, including some bits that you are omitting. I found that document and read it, but it assumes that you have already created something with docs and tests (laid out in an appropriate directory structure). Since I didn't actually submit the package to PyPI, I didn't have to think about twine, but that's another paper cut. Which is my main point: if someone is not already sold on Python, every small quirk and special case becomes another obstacle, and the Python tooling has collected more than a few.

Re:Announcing Pipenv by d0mine on Friday February 03, 2017 @12:01PM (#13561221)
It shouldn't be that hard to demonstrate "shocking" difference in experience.

Could you provide an explicit list of what do you mean by "a breakdown in my earlier comment"? (obviously, I wouldn't ask for more information in my comments if your previous comments were specific enough) e.g., in language X I can do Y using command Z but Python requires A,B,C instead (implying that Z is "shockingly" better than doing A,B,C).

As I understand, the issue is that you want the structure of a project in a language X to be generated for you by a single command (whatever the project) or that this command must be available by default.

I've googled "python create command-line project template":

  $ pip install -U cookiecutter
  $ cookiecutter https://github.com/audreyr/cookiecutter-pypackage.git
I don't remember using this tool but If you like a machine-generated configuration; it might be the start.
Re:Announcing Pipenv by sjellis on Saturday February 04, 2017 @08:11AM (#13566850)
> It shouldn't be that hard to demonstrate "shocking" difference in experience.

Please try installing Rust with the rustup tool, creating a "Hello world" skeleton CLI tool, and compiling a deployable executable for it. It will take you (not joking) perhaps 10 minutes or less if you have never done anything with Rust before. Then look at how you would add docs and tests. The Rust language is not simple, but the tools are really good. Try the same exercise with Go if you prefer.

If you are willing to give this a try and invest the 10 minutes, you'll understand why I didn't use the phrase "shocking" casually: listing off features doesn't really convey how huge the usability gap is.

> Could you provide an explicit list of what do you mean by "a breakdown in my earlier comment"?

It was this:

> "Python best practice: figure out the correct directory structure by reading docs and looking at GitHub repositories, learn how to write setup.py & setup.cfg & requirements.txt & MANIFEST.in files, setup py.test and tox (because Python 2 still lives), write your README in RST format (as used by nothing else ever), and for bonus points: write your own Makefile. Get depressed when you realize that target platforms either don't have Python or have the wrong version."

Re:Announcing Pipenv by bastawhiz on Monday January 23, 2017 @04:51PM (#13466414)
`setup.py` is shockingly awful compared to most other solutions.
Re:Announcing Pipenv by nerdponx on Monday January 23, 2017 @02:25PM (#13465084)
Conda is great software but the Conda/Anaconda ecosystem is a mess.
Re:Announcing Pipenv by pwang on Monday January 23, 2017 @02:52PM (#13465340)
Can you elaborate?