You and the parent are mistaken about the cause of Python’s performance problems. Python isn’t slow because it’s readable—it could be quite a lot faster without compromising readability. Python is slow because it exposes the entire CPython interpreter as the C-extension API, which means they can’t change much about the interpreter without breaking compatibility with some extensions (and they are unwilling to do so). Since performance is so bad, the community leans hard on C-extensions, which worsens the problem.

I suspect one of the reasons packaging is so bad is because for each node on the dependency tree, you need to download and execute setup.py in order to discover the node’s direct dependencies, and to get the transitive dependencies you must download and execute the direct dependencies’ setup.py filles. Since these files aren’t idempotent, we can’t reliably cache the results (these scripts could return different sets of dependencies based on the environment or even the current time or anything else). This at least seems to be the reason all reproducible package managers are slow (by which I mean 30+ minutes to resolve dependency versions for a small non-toy project even if the resolved dependencies at their property versions are already cache to disk).

To clear some confusion: both of you misunderstood my point.

I don't think Python's language readability is the problem.

The problem is CPython's core team shooting down any performance improvement for the sake of keeping the compiler source code readable.

The last two discussions I recall on HN:

https://news.ycombinator.com/item?id=24922522

https://news.ycombinator.com/item?id=11125769

Mine is the top comment on that first link with 34 upvotes. :)

I did indeed misunderstand. Thanks for the clarification! :)

Whl archives ship extracted, cacheable metadata, not the setup script.

Not everything is whl though.

I think you and the first poster are talking about different things. They are talking about readability of the cpython compiler and you are talking about readability of the codebase.

Python has historically refused to implement many optimizations because they reduce compiler readability.

> Python's entire value proposition is that it is "executable pseudocode" (...) so somebody like a scientist (...)

LOL'd hard at that.

What kind of scientist would write "import numpy as np" on their pseudocode? Or multiply matrices with the "@" operator? From the point of view of a scientist, Matlab/Octave, or even Fortran, is executable pseudocode. Numerical stuff in Python seems like an ugly kludge.

As a scientist who had to use MATLAB up until about 2013 because that's what everyone else used, it was such a relief to move to Python. It's true that you can implement a linear algebra routine in a couple fewer characters in MATLAB, but unless that's literally all you're doing, Python is much nicer to work with. The data structures in MATLAB are just a nightmare for general purpose programming, which makes things like loading and parsing data -- things that scientists often need to do -- just terrible. The fact that the notion of the "matrix" (i.e. a 2D array rather than a general ND array) is so deeply baked into everything is a huge headache (a scalar is a 1x1 matrix in MATLAB!). I'm also surprised anyone is particularly bothered by the @ operator. The asterisk for matrix multiplication seems roughly equally unheard of in nicely typeset / handwritten math (you would just write e.g. Ab for a matrix-vector multiplication with no inset operator).

I shudder when I think of my Matlab days, but i’ve programmed long enough in enough other languages to know that we have better options than either Matlab or Numpy/Pandas with respect to API design. We can also have much better performance than Python with many other languages.

I don't know why you got downvoted, but I agree that matrix manipulation in numpy look quite foreign to to somebody new to python/numpy. I had heard this "complain" from several colleagues (mostly former matlab-users). Several colleagues have migrated to Julia, which does a quite good job in producing easy readable numerical code. For instance, the dot-product can be written with the Unicode dot symbol (typed as \dot<tab>)

> matrix manipulation in numpy look quite foreign to to somebody new to python/numpy

Not only to newcomers. I've been using python+numpy for more than 10 years, nearly every day, and it despairs me as much as the first day, if not more. I relish the few moments that I get to write numeric algorithms in any other language.

I think if you zoom out enough Python code looks a lot like pseudo code.

> Those goals are always going to run counter to performance

These days Julia is a very clear counterexample to this sort of claim, I don't think this is true at all.

Julia has a tradeoff w.r.t. startup performance doesn't it?

Also doens't Julia make pretty heavy use of dynamic dispatch, or am I mistaken about that?

Julia's startup time is not short, but it's getting significantly shorter with every release. Right now, the startup overhead is only 0.13 seconds for me.

Some packages still take a while to load, e.g. for me to load the plotting pacakge and produce the first plot takes about 8 seconds (all subsequent plots in that session are fast). This is down from like 30 seconds last year.

One can also bundle heft packages like Plots.jl into their julia system image so they don't have to recompile all that machinery every time they restart julia.

The startup performance issue is just a regular difficult issue, as far as I'm aware. I don't think it has much to do with the goal of "being readable pseudocode" or something. It's getting better with time too.

The "readable pseudocode" kind of code is exactly the sort of code in Julia that you almost always expect to be compiled down to native code quite efficiently. The kind of pseudocode I usually see is either straightforward loops iterating over something where the compiler can infer all the necessary information, or calls to library functions where somebody else has already made sure it's good. I use this a lot in my own code, and, like I said, I don't think there is a tradeoff.

FYI: Poetry (https://python-poetry.org) is actually close to something like npm.

So in my mind, the main things you would need to compete with npm would be:

1. near-universal adoption

2. "just works" experience: i.e. I can clone any random git repo, and run `npm start|build|whatever` and it's always going to work, without having to know anything about my environment or fiddling with configurations.

Does poetry do that?

Poetry wraps around pip, which does have a universal adoption. Which means you can use any package in pypi using poetry.

I don't recall it having custom commands but in my opinion this isn't the job of a package manager.

In most npm projects, you don't depend on C extensions. For python projects that don't depend on C extensions, you already have your "just works" experience with poetry.

If you really want a "just works" experience when dealing with C extensions, you should freeze your development environment one layer up (e.g. VMs, vagrant images, what have you) so that you can always successfully install and compile your C extensions because the underlying system is also kept under version control. And this is independent of whether or not you are working on a Poetry or an NPM based project.

It's not that hard. I suspect that you're just used to simpler conditions.

You'll only really get 2 when 1 is fulfilled.

From my experience, with repos that have Poetry set up, I do get a "it just works" experience that I've previously been lacking when working with python. AFAIK there is no "scripts" section (for e.g. "poetry start" like "npm start") in the config file, or at least no one is using them. You'll have to take a quick peek into the README, but the same is true for npm projects, as the script verbs aren't really standardized.

You can use a "scripts" section in pyproject.toml (it's what poetry reads, and increasingly more Python tools are leveraging it, which is good), but in general each repository/project offers what it wants to offer as commands, nothing in comparison with the "standards in use" for npm and similar. Personally I have used it to give easy access to the "main thing I want to execute" for my own weird projects.

I'm pretty happy using Poetry, and agree with your initial point, partially with the second. There can be some edges with packages involving "anything" binary (to be fair, that is to be expected), and a big issue (for me) is locking the resolution of some libraries, like boto (since botocloud has hundreds of patch releases and the resolver can get pretty crazy unless you play a bit of manual bisection). But this only hits you on poetry lock or poetry add when developing, and only in some cases, I think it's a fair price to pay for a reproducible build.

> AFAIK there is no "scripts" section (for e.g. "poetry start" like "npm start") in the config file, or at least no one is using them.

There is a scripts section in pyproject.toml, which is leveraged by “poetry run”. This is different than for npm, though, since these aren’t dev-environment scripts but the executable scripts that are installed with the project

I usually put https://pypi.org/project/taskipy/ in my poetry dev dependencies to achieve this - then I can just run `poetry run task start`

For the case of Python packages which are primarily exposing a command line script, alluded in the `start|build|whatever`, there is pipx https://github.com/pipxproject/pipx

In my experience npm just works promise brakes as soon as the repo is 1 2 years old.

1 - Nope

2 - Pretty much yeah

Except like pipenv it is dog slow and totally inscrutable when dependencies clash.

It seems a lot faster than pipenv. From the little I've tried, it seems a lot more sane - but I'm biased - I hated pipenv and never saw the point of it.

It used to be incredibly slow but these days I find it's not so bad. It even has parallel pip installs now which while occasionally triggering some issues in projects with over complicated setup.pys is usually a lot faster than a good old pip install -r requirements.txt.

So have you considered the possibility that it may not be the fault of these package managers, but that there may be a different problem underneath it all that a package manager cannot fix for you?

It's a very common misconception that readability is a tradeoff against performance.

The true underpinnings of readability is language malleability, nothing more and nothing less. The whole "dynamic languages are automatically more readable" impression is a misleading consequence of the much more general claim "Malleable languages are more readable", with a malleability = dynamicism substitution.

Malleability is key because readability is really just languages being as close as possible to the problems they are used to solve. This means one of two things

  1)Either the language come preequipped with the concepts and semantics of the problem domain built into their fabric, this is the DSL approach (this will always fail if you aspire to be a "general purpose language", you simply can't hope to match the sheer number of contexts people want to use your language in. One radical conclusion is to abandon the "general purpose language" myth absolutely, and just make all application development language-building, and focus on building the tools to make language building easy.),

  2)Or you make the language malleable and stretchy enough that a tiny handful of the rare programmer-domainExperts breed can construct the whole domain inside the langauge, _with_ _the_ _language_ (no transpiler, preprocessor, etc..., this would require the rare programmer-expert to be an even rarer programmer-expert-languageHacker), then cover up all the low level machinery with syntactic elements that mirror the domain vocabulary. 

That's it. Any langauge that allows you to do the above is a malleable language. Any malleable language is a candidate for being a readable langauge (and a horribly unreadable langauge, if you gave in to irresponsible abstractions). Dynamic languages are malleable because they give you extremely powerful hooks into their semantics, they are very... well, dynamic. The details differ, the two poster childs are python and ruby, they have a grab bag of features that range from operator overloading in python, free form syntax in ruby, and extreme dynamic dispatch and resolution rules in both. The last feature is a common theme in all dynamic languages, and it happens to be a performance killer.

But you absolutely don't have to be dynamic to be malleable, lisps have been doing it since forever, and though lisps are traditionally dynamic, it's macros that make them stretchy, not dynamicism. And there are loads of other mechanims that can make a language malleable without making it unpredictable. It can be as simple as scala allowing unicode in it's source, and as complicated as haskell making lazy evaluation the default and now control structures are just a library. Any abstraction with a "meta" flavor, with hooks extending into the language environment and doing various things depending on how you wield them, makes the language more malleable.

Malleablity is allowing the language's source to take different shapes and semantics according to the whimes of the programmer, performance comes from massaging the source till it fits comfortably into machine semantics without much runtime shenanigans. They are absolutely not in tension, making the code do everything at runtime is just the easy way out.