There's also no reason to believe that startup improvements would make the interpreter incomprehensible; the unstated assumption that improvements in this area must hurt hackability is interesting. IME, optimizations frequently boost both simplicity and performance, usually by unifying disparate code paths and making logic orthogonal.
Saying yes to fixing one thing usually means saying no to all the other things you can be doing with your time instead. Unless you're lucky and can "kill 2 birds with 1 stone".
That is one thing I really want to happen, because I think it begins to open up python to the embedded space. Micropython is nice, but it still needs an interpreter embedded into it.
Cython can use type annotations and type inference to unbox numbers for faster numerical code, but uses ordinary Python objects otherwise. http://cython.org
ShedSkin translates a restricted (statically typeable) subset of Python to C++. https://shedskin.github.io
RPython has a multi-stage approach where you can use full dynamic Python for setup, but starting from the specified entry point, it is statically typed. Since it was created for PyPy, it comes with support for writing JIT compilers. https://rpython.readthedocs.io
Pythran translates a subset of Python with additional type annotations to C++. http://pythran.readthedocs.io
In general, the major hurdle for all attempts to compile Python to native code is that Python code is dynamically typed by default.
All not less dynamic than Python.
What Python lacks is the funding and willingness to actually push one of those implementations to eventually become the new reference implementation.
It will compile python to C++ and then compile that code to machine instructions and calls to the CPython library.
The point is that "better" is almost never a well-defined direction unless you only consider a single use-case. It's almost always a tradeoff, especially in a widely used project.
A language is a point on a landscape of possible language variants, and "better" is a different direction on that landscape for every user.
The problem under discussion is that projects that currently use CPython have slow startup. One potential solution is for those projects not to use CPython. (Certainly it's not the only potential solution, but, a language that tries to be all things to all people isn't going to succeed. Python has so far done an extraordinarily good job of being most things to all people, with "I want native-code performance" being one of the few out-of-scope things.)
I would really like to see some examples where this is the case. Optimizations in my experience have made systems more brittle, less portable and ultimately less maintainable.
For example: as you get to know your use-cases better you might simplify your code to sacrifice unwanted flexibility. Or you might replace a general-purpose data structure with a special purpose one that not just faster, but concretely embodies the semantics your desire.
A case, that is not quite a simplification is removing code re-use. Instead of using function in three different ways, you use three separate optimised functions. Now changes to one use case don't cause bugs in the others. That's the kind of thing that quotemstr meant by "making logic orthognal".
And which is what I mean by making systems more brittle, less portable and less maintainable.
You find a corner case in the original function that's not covered, now instead of fixing it in one place you need to fix it in three places with all the headaches that causes.
So the next maintainer thinks: "Gee I can fix this by bringing all these functions together".
So if they're using an oo language they make an abstract base class from which the behaviour is inherited, or a function factory otherwise.
So now you're back to a slow function with even more overhead, that's even harder to debug.
So the next maintainer comes around and thinks: "Gee I can speed this up if I break out the two functions that are causing 90% of the bottleneck".
Now you have 4 completely independent functions to keep track of.
Repeat ad-nauseum.
MicroPython put together a Python in 250kb. Why the hell can't we make an LLVM frontend for Python that can use type hints for optimization? Sure, you lose some dynamic features as you optimize for speed, but that's the dream. Quickly write a prototype, not caring about types, optimize later with adding types and removing dynamicism.
I'm currently learning Racket and LLVM and I have about 70 more years to live. I'm gonna try make Python fast on slow weekends 'til I die.
There is Cython, of course, which is a great python (+) compiler. I assume you mean a just-in-time compiler as opposed to a static compiler.
The target of my phantasy compiler are microcontrollers. That's why the (correct) comment of quotemstr isn't that big of a concern to me.
There is a lot of stuff out there that goes in this direction. There is nukita (again no small binaries), there is even an abandoned GCC frontend that can compile some minimal examples, but has been abandoned long ago.
Seriously, the time I spent researching this topic - if a proper compiler engineer would spend that on the actual compiler, it'd be done by now.
> time pypy -c 'print "Hello World"'
Hello World
pypy -c 'print "Hello World"' 0.08s user 0.04s system 96%
cpu 0.120 total
> time luajit -e 'io.write("Hello World!\n")'
Hello World!
luajit -e 'io.write("Hello World!\n")' 0.00s user 0.00s
system 0% cpu 0.002 total> Three, the language is not a mere combination of syntax and semantics. Any evaluation should also account for user bases and ecosystem, and in my very humble opinion Lua spectacularly fails at both. I'm not going to assume the alternative reality---Lua has a sizable user base and its ecosystem is worse even for that user base.
> [...] The lack of quality library also means that you are even more risky when you are writing a small program (because you have less incentive to write it yourself). I have experienced multiple times that even the existing libraries (including the standard ones) had crucial flaws and no one seems to be bothered to fix that. Also in the embedded setting the use of snippets, rather than proper libraries, are more common as libraries can be harder to integrate, and unfortunately we are left with PHP-esque lua-users.org for Lua...
I think this critique still holds today, and unless miracle happens (like D), I doubt this is fixable.
Precisely, one of the things that turns me off on Python (coming from Lua), is the unnatural proliferation of different container types.
You might like Tcl - associative arrays (hash tables), arrays (lists), strings, numbers, and code are all the one type (at least conceptually).
PyPy has some great performance characteristics, but startup time isn't one of them.
I would like to note, the cpython ties to the PyObject C abi seem to stymie rather than encourage “hacking”. Cpython seems to have traditionally valued stability over all else.... see the issues pypy has had chasing compatibility with c and retaining speed.
So: normally i’m with you and a language should lean into its strengths, but i’ve always listed startup time as a primary strength of python!
I certainly write and continue to write Python scripts where even an extra half second won't matter. It's doing some manipulation of data where the cost of what it's doing is dominated by loading the data (e.g., grabbing it from some web service), and even if the script is small and quick, it's not so small and quick that I'll notice 50-100 ms being shaved off of it.
Use cases where CPython continues to make sense to me are non-CGI web applications and things like Ansible, where load time isn't sensitive to milliseconds and runtime performance is pretty good. (Although if you believe the PyPy folks, perhaps everything that's PyPy-compatible should be running on PyPy.)
Optimization is very, very rarely completely „free“ - and usually a concious trade of some property for another trait that‘s deemed more important in a specific case.
Simplicity for performance. Code size for compilation speed. Startup time for architectural complexity. UX for security.
For a great product, you need to say „no“ much more often than not. Do one thing and do it well. Be Redis, not JBoss.
I love how this article gets down to the essence of it: https://blog.intercom.com/product-strategy-means-saying-no/
I read that article and I'm still wondering: why Rust?
Are you asking in comparison to some other language? The most obvious other languages in the "compelling better C" niche I think are Ada, D, and Go; Ada and D (I think, I do not know them well) don't have as good of a standard library or outside development community, and Go is less suited to Rust to replacing portions of a process. Go would be a reasonable choice were one writing a VCS from scratch today.
Trying to install a c++ compiler from source (and I've done it several times) is a much less plesent experience, so most people stick with what their package manager provides.
And regarding JS+HTML+CSS, they are still on the stone age of RAD tooling.
Even AWT is better than any option currently natively available to Rust.
After all, "using libraries is no longer a pain" is not what I felt when converting an old toy application from Gtkmm to Gtk-rs.
How so? Is it because Go has a GC and Rust doesn't?
Go has cgo, and has slowly come to a few of the same decisions (e.g., Go gave up on segmented stacks too), and gccgo exists, so it's certainly possible to use Go for this use case. But it's not as suited as Rust.
There's a nice post about calling Rust from Go which goes to lengths to avoid the cgo overhead, even though Rust can just directly expose C-compatible functions and cgo can call them without any special effort on either side: https://blog.filippo.io/rustgo/
More generally a heavy runtime, which creates issues when you're trying to replace parts of a process which has its own runtime, unless you can make the two cooperate (by actually using the same runtime e.g. jvm or graal or whatever).
Go's FFI is also easy to work with but the source of… other issues which is why the Go community often prefers reimplementing things entirely to using cgo and existing native libraries.
Speed and performance sure, but safety automatically rules out C++.
> Want easy multithreading? Go
Want speed, performance, safety, and easy multithreading? Rust.
In context, what I'm really getting at "a sufficiently non-dynamic subset of Python that it can be compiled statically, but also a sufficiently large one that real Python programs can have a chance of being in the subset." PyPy has a thing called RPython that fits the former but not really the latter (I don't know of any non-PyPy-related codebases that work in RPython). In general, adding complete type annotations to a codebase is pretty correlated with making it static enough to do meta-level things on like compiling and optimizing it - for instance if you have a variable that changes types as the program runs, at least now you've enumerated its possible types. It's not the only way of doing so, but it seems to work well in practice and there seems to be a correlation between compiled vs. interpreted languages and static vs. dynamic typing.
I would think developers have the experience to realize this isn't true but I see it all the time on these forums.
And one of the things that affects how productive one hour of work will be - and also whether random volunteers will even show up with one hour of work - is the likelihood of getting a change accepted and shipped to users. This is influenced by both the maintainers' fundamental openness to that sort of change, and any standards (influenced by the maintainers, who are in turn influenced by their users) about how careful a change must be to not make the project worse on other interesting standards of evaluation. It's also influenced by the number of people working on the project (network effects) because a more vibrant project is more likely to review your code promptly, finish a release, and get it into the hands of more users.
So I'm claiming that it's better to spend time on rewriting Mercurial in Rust than to spend time on getting CPython startup faster, because the Mercurial folks are actively interested in such contributions and the CPython folks are actively uninterested, and because there are fewer external constraints in making Mercurial startup faster than in making CPython startup faster. And I'm saying that the more we encourage folks to help with rewriting Mercurial in Rust, the more likely additional folks are to show up and help with the same project, thereby making 1 hour of effort even more productive.
You are literally bringing an attitude of "no" to the question of whether you are being an attitude of "no" to the discussion....
Here are some other things we can say "yes" to:
- Rewrite as much of Mercurial in Rust as possible, which will provide performance improvements well beyond what Python can possibly offer. https://www.mercurial-scm.org/wiki/OxidationPlan
- Spend resources on developing PyPy, which (being a JIT) has relatively slow startup but much faster performance in general, for people who want fast performance.
- Write compilers from well-typed Python to native code.
- Keep CPython easy to hack on, so that more people with a "can do" spirit can successfully contribute to CPython instead of it being a mess of special cases in Guido's head.
Will you join me in saying "yes" to these things and not convincing ourselves to accept mediocrity?