julia> import CSV (takes ~1-2 seconds)
julia> data = CSV.File("/User/orbifold/Downloads/FL_insurance_sample.csv) (takes seconds again...)
I want to explore the data. I vaguely remember there is Gadfly to do that. For some reason it depends on FFTW and a whole bunch of other packages, but the dependency story in Python is equally insane, so whatever. As I'm typing this, I'm waiting for Gadfly to precompile.
julia> @time import Gadfly (104.460307 seconds)
python> import matplotlib.pyplot as plt (~10s if you are unlucky and it has to generate the font cache)
Maybe it's a good idea to do a scatter plot of two of the fields? Let's find out
julia> @time Gadfly.plot(x=data.eq_site_limit, y=data.hu_site_limit) 0.320751 seconds
but that number is not accurate, in fact it took so long for the browser to open the result, that I had time to check the documentation on backends and see whether I'm missing something.
python> %time plt.scatter(data['eq_site_limit'], data['hu_site_limit']) 41.9 ms, a bit more for the window to open
Turns out that is not a useful plot... maybe a histogram?
julia> @time plot(x=data.eq_site_limit, Geom.histogram(bincount=10)) 0.042802 seconds
It in fact again takes several seconds until the browser shows a plot.
python> %time plt.hist(data["eq_site_limit"]) ~100ms
and the window with the plot opens instantly.
People often use Plots.jl or PyPlot which have significantly lesser compile times.
I've had similar problems and so have many of my colleagues in silicon valley. No matter how you spin it and whatever benchmarks you show - Julia is a very slow fast language. :)
The UX of Julia needs major work - everything is just slow. If you've used Python for 10 years and it is like running through molasses. Benchmarks are meaningless for the most part. This is what bothers me about Julia's marketing - it claims to be fast, but in reality, precompilation time alone is a no go unless you're doing high-compute large batch processing. Julia cannot be come a general purpose language unless you fix these issues.
> These "conflations" sound suspiciously like excuses.
I think you're the one conflating total end-to-end time spent between given a task and going home early to see my kids if I used Python.
Obviously, if you are not working on a problem that doesn't need Julia's speed (and makes it worth paying the compilation cost), and you are more comfortable with a different tool, you should use that. You may in fact be better off using Python, or shell scripting or even Excel.
Precompilation time in Julia is akin to `make` in a project with C code. You only compile the library once and use it repeatedly, until you update it. Is it a dealbreaker? Perhaps it is for you (assuming you are using precompilation in the right context). It is not for many who work with Julia day in and day out, and it is something that we continue to improve.
There is something to be said about how easy Python makes development look - despite of the package management warts.
Btw, is there a reason why packages can't be precompiled and even distributed? I am sure you guys have thought about that.
There is work on this. This can be done with https://julialang.github.io/PackageCompiler.jl/dev/, but so far only to a rather large binary / “bundle”
IIUC, In principle it should be possible to do much better if you knew for sure at compile time which methods you would need to dispatch to for the data you ultimately want to run on.
The reason why we can't distribute precompiled .ji files at a package level is because of the way the package resolver works. It depends on the exact versions of dependencies of every package - and even for the same version of an end-user package, there can be slightly different versions of the dependencies installed depending on constraints imposed by other packages.
One major improvement coming in 1.6 is multi-threaded precompilation, and it will leverage all the cores you have.
Of course, any short task that must cold start like a bash script isn't convenient at all (and packagecompiler will generate a fat binary that runs fast, but it's still inconveniently fat if it's for something simple). And Julia's first impression is really important, which the "time to first plot" seems to affect the most (as you need to go deeper on the language to adapt to it's unusual workflow), so I agree that it's a UX priority.
julia> @time Plots.scatter(data.eq_site_limit, data.hu_site_limit) 5.461595 seconds
subsequent calls report significant speed ups compared to matplotlib, but render about as quickly
julia> @time Plots.histogram(data.eq_site_limit) 2.143983 seconds
1. Precompilation time: Only once when you install or upgrade a package.
2. First running time (that includes compilation time): When you call a function like `plot` the first time in a session
3. Second and subsequent running times: The compiled code is now in the cache, and what you see is the actual running time.
Thus what you are reporting here as precompilation (1) is not something that a user will see every time. What they will see is (2) for the first time in a session and (3) on subsequent plots. Could you confirm this is what you see - to ensure that those following along understand the terminology?
Here are the usage times I see when I use Plots on a daily basis in a Julia session:
~ exec '/Users/viral/Desktop/Julia_Releases/Julia-1.5.app/Contents/Resources/julia/bin/julia'
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.5.2 (2020-09-23)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release julia> @time using Plots
11.341632 seconds (20.21 M allocations: 1.108 GiB, 2.87% gc time)
julia> @time plot(1:10,1:10)
2.291147 seconds (3.24 M allocations: 165.569 MiB, 7.80% gc time)
(Another 2 seconds for the window to be drawn the first time on mac)
julia> @time plot(1:10,1:10)
0.000763 seconds (2.80 k allocations: 166.023 KiB)
julia> @time using CSV 1.161851 seconds (2.05 M allocations: 131.757 MiB, 0.46% gc time)
julia> @time CSV.read("/Users/viral/Downloads/FL_insurance_sample.csv"); 4.884074 seconds (6.09 M allocations: 311.878 MiB, 1.90% gc time)
julia> @time CSV.read("/Users/viral/Downloads/FL_insurance_sample.csv"); 0.030916 seconds (147.27 k allocations: 11.361 MiB)