good news is 1.10 already is shaping up to be about 2x faster.

When is 1.10 expected? Just wondering I did a quick google and it seems a lot of the 1.9 improvement were back ported from 1.10? Or is that really 2x on top of current 1.9?

1.10 feature freeze is going to be in the next few weeks. after that it will be a few months depending on how much is required to fix all the bugs that have probably been introduced. 1.9 mostly doesn't shorten loading times (although weak dependencies end up helping a bit). 1.10 has had a bunch of load time optimization which became a lot more obvious once 1.9 got rid of all the stupid stuff. the exact speedups are package dependent, but 2x is a good estimate. some packages get a lot more, some are about the same.

Thanks. Exciting times for Julia!

So ~0.85"?

Python is not compiled and start & load pandas (which is comparable to the libraries loaded in the article) in ~0.4" on my computer, and that's a notoriously slow language.

If I were to use e.g. Rust with polars, load time would be virtually none. And when I have to process ~50k different datasets, I can't afford 0.85" per file, which would translate to ~11 hours of overhead.

It's not 0.85 per file. It's 0.85 for the 50k files combined.

No, because the program has to be run by slurm over several compute nodes, so it can't process them all at once.

as long as you don't have 50k computers, it still should be 0.85 seconds per node which is still tiny.

> it still should be 0.85 seconds per node

No it's not, because I will not architecture my whole pipeline & program around Julia inability to start in a second, I will just use another language.

In good company, that is what Python folks do all the time.

Python does not take 1.7" to load pandas.

> I will just use another language.

Hello C, C++ and Fortran.

> If I were to use e.g. Rust with polars, load time would be virtually none.

Because you're compiling...

And if you need to do the same in Julia, you should also pre-compile or some other method like https://github.com/dmolina/DaemonMode.jl (their demo shows loading a database, with subsequent loads after the first one taking roughly ~0.2% of the first)

Then precompile or use https://github.com/dmolina/DaemonMode.jl.

But 1.7 seconds at first startup isn't even enough time to articulate a serious thought, much less write any good code.

I struggle to believe it's a dent in anyones workflow.

I exposed my use case in the comment chain below; DaemonMode won't help there.

> I struggle to believe it's a dent in anyones workflow.

This is exactly what I dislike in the Julia community, the ‶if I don't have this issue, then the ones meeting it are holding it wrong″ attitude.

Perhaps we got off on the wrong foot here as I was a bit aggressive, my apologies for that. I see your use case below as you said, and just want to (hopefully politely) explain that your opinions on this startup atency are still likely incorrect.

I have no illusions that I'll convince you - I don't think anyones ever done that on the internet :) But for educational purposes and anyone else interested read on. Also to be clear - "performance" is probably not the number one you'll see people advocating strongly for Julia. It's moreso because if you're in a position where you're writing code that is being run on 50K databases, and not editing CSS styles - what you're doing fundamentally, at some point under the hood, is mathematics. And the flow and design of the code should reflect that. Julia gets us a lot closer.

A few things before I mention your use case specifically:

Running

    $ julia myprog file_$x.db

in a bash script, 50k times would indeed take that 0.85 seconds per run and would be a ridiculous amount of overhead.

But it would would be the equivalent of the bash script in rust (and actually quite similar, as at the end of the day they both target LLVM)

    $ cargo build --release
    $ ./target/myprog file_$x.db

## Solution 1: Precompiling to reduce startup latency

But obviously you wouldn't compile each time - you'd compile once and distribute the binary, or pre-install the compiled libraries beforehand, whatever. In rust you can obivously easily statically build things and distribute. In python you're just calling pre-built libraries in other languages anyways.

In Julia that's not so easy because we have to carry around the GC, compiler, and even a patched LLVM. (See Why is my "Hello World" executable 300MB?!" posts on PackageCompiler.jl....). But Python obviously has the same issue. Your first solution here is to just precompile all of your libraries and code

With a naive approach:

    $ time julia -q noncompiled_prog.jl                                 
    1.57s user 0.14s system 177% cpu 0.961 total

Using a compiled sysimage with our library:

    $ time julia --sysimage=DataFramesSysImg.so compiled_proj.jl
    0.16s user 0.21s system 21% cpu 1.691 total

## Solution 2: Julia already plays nicely with distributed computing Distributed tasks like you're talking about are literally baked into the language, with macros making it trivial to work with clusters, and yes there are people using it with Slurm. For example...

    @parallel for i=1:N
        someCoolFunction(args)
    end

to parallelize across your cluster.