CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

Re:CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster tha... by fermienrico on Monday October 12, 2020 @12:13AM (#24751324)

Thanks for the insight. I think still for most engineering development - its mostly the one-off launch time that predominates. This is exactly why I don't use C++ for general purpose use.

There is something to be said about how easy Python makes development look - despite of the package management warts.

Btw, is there a reason why packages can't be precompiled and even distributed? I am sure you guys have thought about that.

[ Parent | on HN ]

Re:CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster tha... by cbkeller on Monday October 12, 2020 @03:33AM (#24752227)

> precompiled and even distributed

There is work on this. This can be done with https://julialang.github.io/PackageCompiler.jl/dev/, but so far only to a rather large binary / “bundle”

IIUC, In principle it should be possible to do much better if you knew for sure at compile time which methods you would need to dispatch to for the data you ultimately want to run on.

[ Parent | on HN ]

Re:CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster tha... by ViralBShah on Monday October 12, 2020 @09:34AM (#24754172)

Yes, you can use PackageCompiler for your collection of packages. We are also exploring doing this in the background when you add more packages.

The reason why we can't distribute precompiled .ji files at a package level is because of the way the package resolver works. It depends on the exact versions of dependencies of every package - and even for the same version of an end-user package, there can be slightly different versions of the dependencies installed depending on constraints imposed by other packages.

One major improvement coming in 1.6 is multi-threaded precompilation, and it will leverage all the cores you have.

[ Parent | on HN ]