What people decry about “the Python ecosystem” is simply a function of its success as a generalist computing language over 25 years. Nobody uses Typescript to build Linux distributions; nobody uses Go to build lib-gluing GUI apps; nobody uses Rust to write spreadsheet formulas. Python does all that (and more), and this of course resulted in a sprawling community, with tools pulled every other way. If any of your new shiny languages will ever achieve a similar level of popularity, you can bet their ecosystem will also become a complete mess.
That's because Python is old. It's as old as Haskell. It's older than Java and JavaScript. It's older than the Borland Delphi and C++Builder IDEs. When Python was released, C++ was only 5 years old. Python is older than Linux. Rust and Go are both older than C++ was when Python was released. Hell, when Python was released, Perl was only 3 years old.
Further, Python's age doesn't mean that we can't or shouldn't criticize it. You shouldn't get a free pass on what you could do better just because you've done other things right. We should absolutely point to what doesn't work well and lobby for improvements even if they require significant overhauls. While, yes, Guido and company have worked very hard on developer experience out of the box, it would be really nice if someone would care about system administration and the long tail, too. It's great that your engineers love your tools, but someone still has to live in the homes that they build.
It's not like anybody here is shocked or confused when they hear people complain about Python versioning being a rats nest. I'd wager we've all experienced it at least once just like we used to have problems with multiple concurrent versions of the Java Runtime Environment being installed or other fun forms of dependency hell. If virtualenv or conda are such good solutions, why do so few projects seem to use them or deploy with them? If popularity or ubiquity are the measure of what's good in the ecosystem, doesn't that suggest that there's still a problem to be solved? A language should direct programmers to styles of system design that makes system management easy and clear, and Python does not do that at all. Is there a problem with complexity? Is there a problem with lack of education?
I don't disagree, but for people paying attention, it's become really boring. "I've just switched from $language to Python [because work told me to], and this is what I hate" is almost a parody, at this point. It's a sign the ecosystem has reached ungodly proportions.
> If virtualenv or conda are such good solutions, why do so few projects seem to use them or deploy with them?
I don't know about conda, but venv/virtualenv was the deployment standard before Docker happened (I'd argue it still is, for people who won't/can't use containers). Personally, I still like to use venvs even in docker.
> If popularity or ubiquity are the measure
How do you measure popularity? So often the pip hate seems to come from a very loud minority. Meanwhile, the ecosystem continues to expand and people keep using pip/venv without any problem, because it works for most cases (now that wheels have fixed the "can't build on Windows" issue) and it's simple enough. Somebody upthread compares pip with Maven, and it makes me laugh: the Java ecosystem is shrinking and the Python one is exploding, and stuff like the user experience of Maven vs pip is among the reasons.
> there's still a problem to be solved
Of course there is, there is always one; I'm sure that you'll find Cargo critics too, once that community grows enough, and there are plenty of loud NPM haters. Trying to be all things to everyone takes careful reasoning and herding - because it's a political problem as well as a technical one, in a polis that keeps growing. Shouting PIP SUCKS!!111!! is only going to result in more crap like pipenv. The shortcomings of pip have well-known for a while, but the solution is not trivial, as pipenv demonstrated.
> A language should direct programmers to styles of system design that makes system management easy and clear
That's like, your opinion, man. One of the strengths of Python is that it's just as structured as you want it to be, and no more - even when that might not satisfy someone's arbitrary definition of a Platonic application.
Further, if you factor in the growth of languages like Kotlin and Scala the JVM is blowing Python out of the water.
The Java module experience is miles ahead of the Python one. There's no global module path where you can install different things on different machines, nothing that screws up if you accidentally run it as root, no stateful virtualenvs where different shell windows run the same project with different versions of python. You just list your dependencies in your POM (which, yes, is XML; it was the early 2000s, everyone was doing it, and it does make it easy for your IDE to put a nice editor interface on it) and that's it, you're done: released versions are immutable and transient dependency resolution is deterministic, so you don't have to worry about pinning/unpinning or anything like that. There's a well-known repository that all the important libraries are in; as and when you want to have a company private repository there are a couple of standard ones you can pick from and run, either just for your own artifacts or proxying third-party libraries from the central one as well. It all Just Works.
Maven came out 14 years ago and Python still doesn't have anything that works nearly as well.
You are seriously comparing Maven, a huge and over-complicated xml-based system that is not even installed by default and that people hate so much that there are umpteen alternatives (gradle etc), with `pip install -r requirements.txt` that works out of the box? I just can't even...
> Python still doesn't have anything [like Maven]
And I thank the Gods for that.
It's not complicated. It's verbose, that's the nature of XML, but the behaviour is very direct.
> is not even installed by default
Which is the better approach because it means it's not coupled to specific versions of the language itself. You can use a single maven install to manage multiple versions of Java (or vice versa); if you need to rely on a new maven feature in a project that's stuck on an old version of Java, it's no problem.
> people hate so much that there are umpteen alternatives (gradle etc)
There will always be refuseniks (particularly for pioneers; most of the things people hated maven for are the same things people love cargo et al for) but the Java ecosystem has done a very good job of keeping everything interoperable; it doesn't matter if one of the libraries I call builds with gradle, because there's a common packaging/dependency/repository standard, so everything will just work exactly as it should.
> with `pip install -r requirements.txt` that works out of the box?
Until you come to upgrade, or until you run it the wrong way (e.g. forgetting to start your virtualenv first) and mess up your system install.
Works out of the box until you're missing a distro package that is required to build a dependency that needs to be compiled from source.
I've never used maven so I don't know if it is better or worse, but I am not a fan of languages having their own package management system that has not integration with the distro one (which probably also offers some of the same packages, and mixing them breaks things in subtle, annoying ways).
With decently-maintained packages on PyPI, i.e. shipping prebuilt wheels for the most common architectures, that's no longer a problem - unless you insist in using the distro package-manager, in which case you should pick it up with the distro people. If you stick to pip+venv, on most common architectures, these days chances are you only need a simple `install`.
> I am not a fan of languages having their own package management system that has not integration with the distro one
You must feel miserable then, considering it is pretty much all modern ones. Go, Rust, Node, Perl, Python, PHP, Java, even C# and friends...
There will always be tension between what developers want (the library released yesterday and can be forever tweaked) and what OS/sysadmins want (the library that has been tested for months and can be locked down). This is why platform-agnostic delivery systems for developers are so popular, and it is not going to change any time soon.
Mixing is where Python (and Perl, even though it is integrated with the distro package management) go wrong, IME. Maven is isolated from the host package management but completely (or at least completely enough, in practice); most OSes don't bother trying to ship system-wide versions of Java libraries. The OS manages the JVM (and Maven), Maven manages whatever libraries an app wants on a per-app basis, and neither interferes with the other.
This does mean you get very little help managing dependencies of JVM libraries on native libraries; fortunately those are rare enough in the JVM ecosystem that you can handle the few that do occur by hand, IME.
It's about the same for packages with a native component. Local build tools are still needed.
Java stuff is slightly less likely to have a compiled component in the first place, in my entirely subjective impression. Maybe that's because of convention, or performance; I don't know. But when a compiled dependency does exist, and nobody included a prebuilt chunk of binary for your architecture in the jar, a build is needed in roughly an equivalent fashion to what pip (or npm, cpan(1), etc.) do.
https://en.wikipedia.org/wiki/Lightweight_Java_Game_Library
Whereas pyglet, seems to me like a high-level regular game engine that includes a widget library.
If a package owner distributes a wheel, you're good. Most packages do now.
> there are umpteen alternatives (gradle etc)
there was ant (build tool) then came maven which is much more flexible and contains dependency management capabilities (and _is loved_ by many) nothing to add that the parent did not already say. Then came gradle which has faster build speed and features like incremental builds and a Groovy based configuration.
pip did not ship with python for a long time, and has problems of its own.
You make it sound like adding Apache Groovy for configuration to a build system is an improvement. When build systems with a declarative config language, such as make, ant, and maven, replace procedural build scripts, that's the improvement. Using a procedural language like Groovy instead is a large leap backwards because it encourages programmers to add unneeded procedural features to build scripts, creating an unmaintainable mess.
> a huge and over-complicated xml-based system
It was an attempt to show that there are not 'umpteen different alternatives' and the ones that exist, do so for a reason. Though looking back, I should have worded it in a better way.
A post on my blog:
pip now installed with Python 2.7.9:
https://jugad2.blogspot.com/2015/03/pip-now-installed-with-p...
which mentions:
https://docs.python.org/2.7//installing/index.html
and::
https://pip.pypa.io/en/latest/installing/#pip-included-with-...
I've used Python versions before that, where you had to install pip separately, and it was not trivial (to find it / install it), although not very difficult either.
It has some performance issues but it's still quite a recent project.
Is that true? I see a lot of pom.xml files specifying version ranges for dependencies. If I have a project specify a dependency that has its own dependencies with version ranges specified, then there doesn't seem to be an easy way for me to pin all the subdependency versions. (Yarn and npm both generate a lock file automatically, pinning all dependency and subdependency versions, which seems great.) It seems like I could specify all of the subdependencies in my own pom.xml file too with specific versions, but maven doesn't appear to make that easy to do.
Version ranges are indeed nondeterministic, that's why they're discouraged by the community (and IME very rare).
Pinning specific ones is easy enough (I tend to just look at the dependency tree in eclipse and right click -> lock transitive dependency version); I can't find a way to do that for all transitive dependencies though (I've never had more than a handful of transitive dependencies that used ranges, so it's not been a problem I've had tbh).
Today, if you are interested in a reproducible build environment, you will use pyenv to locally install an interpreter and stdlib, and pipenv to locally (to your project) install libraries and dependencies.
Rust is doing some interesting things with Rust 2018 Edition, where there is a complete system harmonization point. If it takes off there, I'd expect other languages to pick it up.
My question though is a bit different: Can Python catch up?
I know - strange question, but momentum is like that. It's often hard to see something going faster than you until it passes you. Perl is probably the best example, where it was way out front, and the PHP, which was way behind, but going mach 10, passed it quickly in the web space. Perl never caught up, and has lost significant relevance compared to the stature it used to command.
Consider; the reason for installing dependencies locally to your project is that it allows you to be precise about what library version you want. For example in Rust, we have Cargo.toml and I can have dependencies like primal = "0.2". I hate this model. It takes us right back to the old days of projects shipping whatever version of a library they want and never updating it. For small projects this is a constant source of security issues and code stagnation.
I groan internally every time I see a project I'm interested in is written in node. Not because I dislike the language, but because I know when I download and build it there are going to be a dozen outdated dependencies hardcoded in the config file, some with "severe" security warnings flagged by npm.
I advocate letting library developers and distribution maintainers do their jobs. Most of the time new versions of libraries are supposed to be backwards compatible. When they're not the distribution can ship multiple versions of a library and make sure they all have security updates backported.
There are cases where managing all your own dependencies is important (like closed source web applications), but I think those are exceptions. Dependencies were supposed to be a solved problem.
It evolves constantly. Python packaging then and now has massively changed (largely without breaking compatibility, mind you). It's not perfect, but it is much better than it was in 2005.
I know pythons big thing is backwards compatibility, and breaking things is almost the biggest sin in that community (python2...), and maybe I’m ignorant, but I just don’t see how implementing a single standard package manager with a standard manifest and lockfile would break anybodies anything. All they have to do is make it so pip still works without it, and if they want to see how to do that, just look at NPM, who before this year was notorious for errant package versions due to the lack of lockfiles, but now they have it and I think pretty much everyone was happy about it. They literally could just port bundler to Python and call it a day, the code is open source, they can look at how it works, and it’s one of the most well regarded package managers out there, I’ve maybe run into 2 issues ever with it. The only other package manager I see get more praise is cargo, which they could (should) also look at for inspiration.
I know things have been improving. When I was using Python daily, the standard was still punching in some incantations to start of Virtual Env (which still confuses the hell out of me to this day) and piping the contents of a requirements.txt file into pip (with I think at least some level of version locking). But, from what I’ve gathered is there are some solutions being built to bring pip out of the early 2000s, but they’re quite fragmented efforts, and they all fall short in different places.
Python is a great language, pip and all the bullshit that comes with it is terrible.
Now that I’ve wrote that out, I’m wondering if packages can even pin their deps required versions. I’ve only published one python package years ago, and I can’t remember if that’s possible. Which I shouldn’t have to even worry about in a modern package management system.
I know the above example wouldn’t be possible in any of the systems I mentioned above, because the aside from installing packages, they ensure that the versions of every package are compatible. That’s why lockfiles are so important, assuming you’re downloading decent packages, the package manager can assure you that they’re all going to work together, also allowing you to update without having to worry about some random deeply nested dep isn’t going to break some other package when it gets updated, making updating (usually) a breeze.
In pips current state, it seems more or less like a slightly more strict NPM before yarn made them get their shit together (still vastly prefer yarn). The only difference is your packages are wherever Venv puts them instead in a “pip_modules” folder in the project, which is at least better than a global free-for-all. Though, that also might be less of a headache than dealing with Venv, I hope it’s gotten better since I’ve used it, because that was always such an annoyance.
> Now that I’ve wrote that out, I’m wondering if packages can even pin their deps required versions. I’ve only published one python package years ago, and I can’t remember if that’s possible.
It is.
Seems to be active up until around July-August
And no one could possibly argue that Python hasn't had irregularly poor leadership for a long while. I'd go so far as to say that its success is in spite of its leadership, culminating with Guido basically giving the middle finger and walking away this year. Can't say I blame him.
It is the nature of newer languages to learn from the mistakes of past ones. But the big irregularity which wasn't learned is that these languages I listed have very strong leadership. In the case of Go, I'd argue the strongest leadership of any language ever made, often to the disappointment of the community.
However, there are obvious costs in terms of context-switching, learning curve, career flexibility, and so on, which is why general-purpose languages are so popular.
Java has had Eclipse right from when users needed it for prime time.
And yes, Python's repl is barely a REPL. You are better of writing code and running it. Its not exactly Lisp experience.
And I regularly run into issues with people using non-distribution Java packages that don't really integrate as well with the OS as the packaged one would.
I feel like some of his complaints about the ecosystem there were really just complaints about his specific setup.
really?
which way?
embedded? fastcgi native? slowcgi? fpm? cli?
natively managed or via process control?
which process control?
which ini file is configured for which application again?
are my suexec/whatever wrappers working?
if not, which user/group/permission N levels up is causing the issue?
which bytecode cache to use? is that working properly?
why aren't my URL rewrites working properly?
The real problem here isn't that Python was used to mean the language alone when it should have been used to mean the entire ecosystem, or vice versa. The problem is that it's ambiguous. I didn't mention that points 1 or 2 are about the ecosystem to invalidate them. There are stronger refutations available. I was just trying to identify a scope that the author had left ambiguous.
This argument really summarizes a beautiful and dangerous thing we see in the tech community far too often; you have a strong technical and scientific understanding of the system, but lack product and design thinking.
You're right. Its a problem with the ecosystem, not the language. I'm still not going to use python because of the ecosystem. If the language cares about its users, it needs to care about the Product, which includes the ecosystem. Users interact with the Product, not just the Language.
That's something recent languages like Go, Typescript, Rust, and even old languages like Java, understood quite well. Its not just about the syntax and the compiler; getting users into your language means caring about the IDE, caring about the development experience, the package management, the libraries, basically everything that a developer will touch. That doesn't mean you have to touch all that stuff as the core development team, but it does mean you need a Strategy and a Message.