Nuikta (http://nuitka.net/) already does that and much more:

- it compiles your program and make it stand alone so you can distribute just the exe

- it makes it start faster

- it makes it run faster

- it's fully independant of the system python. Actually your system doesn't even need a python at all

I don't get why it's not used, it's very robust, compatible with 3.6 and on some of my script I get about x4 speed up just on start up alone.

This is different from the package that I've tested (PyInstaller or py2exe).

Is Nuikta compatible with numpy, pickle, etc? I remember that numpy was very problematic with compilers like pypy for a long time.

In my experience it's easier and more reliable than PyInstaller or Py2exe to use, and cross plateform (but no cross compilation). It doesn't pack python files with an executable. It translates the Python code to C then compiles it.

Nuikta supports numpy officially, you can even see it in change logs: http://nuitka.net/posts/nuitka-release-0521.html

I haven't tried pickle.

>This is different from the package that I've tested (PyInstaller or py2exe).

In an ancestor comment you say:

> A single file several hundred Megabytes in size.

Are both points referring to PyInstaller? Asking because I've tried out PyInstaller with small CLI as well as GUI (wxPython) programs, and the resulting EXEs did not reach near that size, IIRC.

It was around 150..300 MB if I remember right. Admittedly, I do import a lot of modules. Program startup for me is in the seconds, not milliseconds. I maybe could cut it down 50% or even 80% but then that would cost a few weeks or a few months. Having a quick and robust solution to be implemented in a single week to cut down startup times by that amount would be highly preferable.

Interesting. I see what you mean. That tip that someone else mentioned, IIRC, in this thread, to import modules inside functions that use them, could cut down startup time, but only if those functions are not called at startup, only later during the program's run. But it would not change the EXE size, I guess.

I wish Python and all other interpreted languages came with a way to build EXEs from the start. It would be great for deployment.

First time I hear about this, and I've looked for alternatives to cxfreeze and its cousins in the past.

Any time I see something like this, I feel like I'm hearing about some homeopathic cancer cure. If Nuitka actually does what it says it does, it's solving a big recurrent problem for the Python community, so why is nobody talking about it?

Having followed Nuitka since it started, I can offer my perspective:

- Before Nuitka, someone already did a "Python->C" compiler along similar lines: translate the Python source code into the C calls that the interpreter will make, eliminating any interpreter overhead and providing C compiler optimization opporunities. That thing sort-of-worked (with v1.5.2 IIRC), but was cumbersome to use and delivered a meager 5% performance improvement for the cases it did support; it was abandoned.

- Nuitka's plan had the same thing as a starting phase; people told the Nuitka guy that he's wasting his time based on prior experience. When he actually delivered a mostly-robust working version (much more usable than the previous attempts ever were), it indeed delivered only a small performance gain compared to CPython.

- As a result, it seemed like the community believed both that the whole thing is futile, and that the developer is fighting windmills.

- a lot of time passes, Nuitka keeps improving with better analysis, translation, compilation, etc - but the community has already cemented its opinion.

- Nuitka remains a useful magic system known to few.

I would say that the early Nuitka versions (and the prior attempt) gave it a SEP field that has never been lifted, and short of e.g. DropBox or Facebook adopting it, nothing will lift it either.

I think Nuitka is an answer looking for a problem. Usually people move away from Python. If they really need more low level performance, or implement whatever system in Python itself, to solve the problem Nuitka might solve for them.

That said, it's a wonderful project, I hope more and more people will find it useful for them.

Quite the opposite, distributing a Python binary is one of the most popular demand in the community, along with better multiple core and a JIT.

It used to be packaging and V2/V3, but those are fading away now with wheels everywhere and 3.4+ being the new love affair. Python has been in the habit of improving every year, steadily for 28 years, solving the problems the community asked every time.

How is that the opposite? :)

It just means that there are a lot of folks who stay on Python, but want better deployment. That's great, but we don't see those who simply move away from Python (and use Electron with a Rust backend for example, or go full web + maybe native Android/iOS apps).

That's the question I'm asking.

Not only it's a beautiful tool, but the author has been quietly and steadily working on it for 8 years. Compatibility is the number one goal.

The guy has a lot of rigor and humility, so maybe communication suffered ?

Let's give it some visibility then. https://news.ycombinator.com/item?id=16980704

hg allows loading modules at runtime. maybe thats a problem.

I doubt it, since you can pass pass manually a list of all modules you want nuikta to embed with --recurse-plugins=MODULE/PACKAGE

But the problem is that you can't know the list you need at embed time for hg, because extensions are arbitrary python files discovered at runtime.

I collect ideas, especially weird and powerful ideas.

I've learned not to try to talk about it because of that question: "If foo is so great, why isn't everybody using it?"

It's one of the single greatest frustrations of my life. I don't know. I've never made any progress on it. The best you can say is, "Well, that seems to be human nature." The world is full of "magic beans" and most people seem interested in banging their heads against the wall.

(Did you know, you can make a 140hp engine that fits in the volume of two stacked pizza boxes and has only one moving part?)

Anyhow, Nuitka is great, it does do all that.

And the creator is a freakin' saint for putting up with the way he's been treated by the Python community, is my opinion.

Also, you should write a blig about those ideas. Or a place where we can share it, but in a non tin foil way / silver bullet way.

Eg: after 7 years of having malaria, one tropical disease doctor explained to me that we have been able to cure malaria for years. Generalist doctors usually don't know it because they don't encounter the disease often enough to keep up to date. It's kind of hard pill to swallow given that i always though you had it for life.

I'd love to, but from experience I can tell you, you just get skeptics and crackpots and scammers and suckers crawling out of the woodwork and gumming up the scientific/inventive process. That, combined with the apathy of the general public, means that it's just hard to get a real conversation going in a "non tin foil way / silver bullet way".

It also means that a lot of great ideas go nowhere, or take 50-100 years to get adopted. Your experience with malaria is an example. My condolences btw, that sounds terrible.

In any event, there's Rex Research: http://rexresearch.com/1index.htm (IGNORE HOW IT LOOKS!!!) This fellow has been collecting inventions and other weird stuff for decades, since before the internet. He used to run little ads in the back of Popular Science and others like that. Yes, the site looks a little... creative, and much of the stuff he lists is just crackpottery, but the stuff that isn't is mind-blowing.

Just one example, one of my favorite devices: the Hilsch-Ranque Vortex Tube. It's a "Maxwell's Demon" (although it does not violate Thermodynamics. Of course.)

> The vortex tube, also known as the Ranque-Hilsch vortex tube, is a mechanical device that separates a compressed gas into hot and cold streams. It has no moving parts.

http://rexresearch.com/ranque/ranque.htm

You can actually buy these to go on the end of a compressor and provide "spot cold" for cooling off whatever. I emailed a company that sells them once to ask what would happen if you set it up in a feedback loop so that the cold output was chilling the input line, but they were uninterested.

Oh hey, time marches on and it now has a wikipedia page! https://en.wikipedia.org/wiki/Vortex_tube

Anyhow, like I said, I'd like to blog about this stuff but most of the people who would be into it would either haters or credulous fools. Speaking of which, youtube has lots of videos of people talking about and sometimes even demonstrating things. But again, you have to wade through all kinds of bullshit and scammers and hoaxers and skeptics and credulous fools to actually find the handful of people who take this stuff seriously but can maintain a proper detached scientific attitude to actually investigate it. YMMV

Saint is the word given he has be working really seriously at it for 8 years, alone, never complaining, and giving away everything without any recognition.

Bingo.

It's a function of how many people really need that solution. It's high enough that it exists. It's low enough that it's not a thing you do by default or even talk about much. But if you need it and it's compatible with your code - it's there.

Actually any sysadmin or scripter wannabe could benefit from nuikta. Twitter even invented pex because of that. Nuikta took some time to become 100% compatible with the latest versions of cPython, but now it's the case so let's all enjoy it.

I like nuitka and what it's doing, but it's just not that comfortable in every single situation. There's value in seeing the source and being able to modify it in place without a compile&redeploy steps. There's value in dtrace support which as far as I can tell nuitka doesn't have. And other small things like existing profilers which work with cpython stack traces specifically.

So if there is a reason to use it - great. But that's not every situation.

Can't you use CPython for development and Nuitka for deployment?

Does nuitka build a static executable or do you still need to supply shared libraries with the executable?

It builds a static executable if you pass it the --standalone option.

No, it doesn't.

  $ echo 'print "Hello world"' > hello.py
  $ nuitka --standalone hello.py 
  $ ldd hello.dist/hello.exe
  	linux-gate.so.1 (0xf7f94000)
  	libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xf7895000)
  	libpython2.7.so.1.0 => /home/jwilk/hello.dist/libpython2.7.so.1.0 (0xf7508000)
  	libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xf732f000)
  	/lib/ld-linux.so.2 (0xf7f96000)
  	libz.so.1 => /home/jwilk/hello.dist/libz.so.1 (0xf7310000)
  	libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xf72f1000)
  	libutil.so.1 => /home/jwilk/hello.dist/libutil.so.1 (0xf72ed000)
  	libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0xf71eb000)

Ok, it buids a static executable against the python runtime and extensions, including the c ones. It doesn't against libc, lubutil, etc. But aren't those almost always installed ?

If it's a real problem on your machines, it's PR time !

Nuitka has not been able to compile any Python code I've written myself. It's not used because it's incredibly limited.

When was the last time you've tried it?

Is it a perl2exe descendant, packing the interpreter into an executable wrapper?

No it compiles python to C, then compile the C.

I know this isn’t everyone’s favorite but Cython has a way to convert your python code Into an executable with Python embedded and I bekievr it also Packs your imports

Cython is a complicated beast but I feel like it just needs a more friendly wrapper for this to be more widespread.

https://stackoverflow.com/questions/22507592/making-an-execu...

https://github.com/cython/cython/wiki/EmbeddingCython

Why Cython isn’t in the stdlib (I think it could easily replace ctypes) is beyond me sometimes

I worked on one Python application that had a startup time problem because it was on a network filesystem with slow metadata/stat times. It took several seconds to start Python.

We were able to solve most of the problem by zipping up the Python standard library and the our application.

That is, if you look at sys.path you'll see something like:

  >>> sys.path
  ['', '/usr/local/lib/python36.zip', '/usr/local/lib/python3.6', ...]

If you zip up the python3.6 directory into python36.zip then it will use that zip file as the source of the standard library, and use the zip directory structure instead of a bunch of stat calls to find the data.

This should also include getting access to the pre-compiled byte code.

You can also have Python byte-compile all of the .py files in a directory as part of your build/zip process.

  python -m compileall --help

Don't forget

  find . -type f -name "*.py" -delete

right afterwords.

Also note calls to imp.load_source need to change to imp.load_compiled, and any .py files references directly in code need to be changed to .pyc (this is with 2.7, not sure about 3.x)

Thanks, I will try that!

cool!

I think design choices made in Python simply don't allow for comprehensive ahead of time compilation. For what it's worth, they have recently landed snapshots in Dart that do what you want:

https://github.com/dart-lang/sdk/wiki/Snapshots

It's what Flutter uses on iOS since you can't run JITed code; AOT compile it and load it as just another shared library.

Check out subpar: https://github.com/google/subpar

also it’s not static linked, so you need to make sure all of the shared libraries exist on the host, requiring to install a whole bunch of trash.