First import took >10s just to precompile and then CSV.File took 5.21s to read the file. A fresh pandas install takes ~10s to import and then reads this file in 81.2 ms ± 1.25 ms. CSV.jl takes 38 ms on a second run so it is twice as fast, however especially to someone that just starts using Julia, these >100x slowdowns constantly happen. Especially when using libraries that heavily rely on multiple dispatch specialisation for performance.
julia> @time using CSV 1.161851 seconds (2.05 M allocations: 131.757 MiB, 0.46% gc time)
julia> @time CSV.read("/Users/viral/Downloads/FL_insurance_sample.csv"); 4.884074 seconds (6.09 M allocations: 311.878 MiB, 1.90% gc time)
julia> @time CSV.read("/Users/viral/Downloads/FL_insurance_sample.csv"); 0.030916 seconds (147.27 k allocations: 11.361 MiB)
julia> import CSV (takes ~1-2 seconds)
julia> data = CSV.File("/User/orbifold/Downloads/FL_insurance_sample.csv) (takes seconds again...)
I want to explore the data. I vaguely remember there is Gadfly to do that. For some reason it depends on FFTW and a whole bunch of other packages, but the dependency story in Python is equally insane, so whatever. As I'm typing this, I'm waiting for Gadfly to precompile.
julia> @time import Gadfly (104.460307 seconds)
python> import matplotlib.pyplot as plt (~10s if you are unlucky and it has to generate the font cache)
Maybe it's a good idea to do a scatter plot of two of the fields? Let's find out
julia> @time Gadfly.plot(x=data.eq_site_limit, y=data.hu_site_limit) 0.320751 seconds
but that number is not accurate, in fact it took so long for the browser to open the result, that I had time to check the documentation on backends and see whether I'm missing something.
python> %time plt.scatter(data['eq_site_limit'], data['hu_site_limit']) 41.9 ms, a bit more for the window to open
Turns out that is not a useful plot... maybe a histogram?
julia> @time plot(x=data.eq_site_limit, Geom.histogram(bincount=10)) 0.042802 seconds
It in fact again takes several seconds until the browser shows a plot.
python> %time plt.hist(data["eq_site_limit"]) ~100ms
and the window with the plot opens instantly.
People often use Plots.jl or PyPlot which have significantly lesser compile times.
I've had similar problems and so have many of my colleagues in silicon valley. No matter how you spin it and whatever benchmarks you show - Julia is a very slow fast language. :)
The UX of Julia needs major work - everything is just slow. If you've used Python for 10 years and it is like running through molasses. Benchmarks are meaningless for the most part. This is what bothers me about Julia's marketing - it claims to be fast, but in reality, precompilation time alone is a no go unless you're doing high-compute large batch processing. Julia cannot be come a general purpose language unless you fix these issues.
> These "conflations" sound suspiciously like excuses.
I think you're the one conflating total end-to-end time spent between given a task and going home early to see my kids if I used Python.
Obviously, if you are not working on a problem that doesn't need Julia's speed (and makes it worth paying the compilation cost), and you are more comfortable with a different tool, you should use that. You may in fact be better off using Python, or shell scripting or even Excel.
Precompilation time in Julia is akin to `make` in a project with C code. You only compile the library once and use it repeatedly, until you update it. Is it a dealbreaker? Perhaps it is for you (assuming you are using precompilation in the right context). It is not for many who work with Julia day in and day out, and it is something that we continue to improve.
There is something to be said about how easy Python makes development look - despite of the package management warts.
Btw, is there a reason why packages can't be precompiled and even distributed? I am sure you guys have thought about that.
There is work on this. This can be done with https://julialang.github.io/PackageCompiler.jl/dev/, but so far only to a rather large binary / “bundle”
IIUC, In principle it should be possible to do much better if you knew for sure at compile time which methods you would need to dispatch to for the data you ultimately want to run on.
The reason why we can't distribute precompiled .ji files at a package level is because of the way the package resolver works. It depends on the exact versions of dependencies of every package - and even for the same version of an end-user package, there can be slightly different versions of the dependencies installed depending on constraints imposed by other packages.
One major improvement coming in 1.6 is multi-threaded precompilation, and it will leverage all the cores you have.
Of course, any short task that must cold start like a bash script isn't convenient at all (and packagecompiler will generate a fat binary that runs fast, but it's still inconveniently fat if it's for something simple). And Julia's first impression is really important, which the "time to first plot" seems to affect the most (as you need to go deeper on the language to adapt to it's unusual workflow), so I agree that it's a UX priority.
julia> @time Plots.scatter(data.eq_site_limit, data.hu_site_limit) 5.461595 seconds
subsequent calls report significant speed ups compared to matplotlib, but render about as quickly
julia> @time Plots.histogram(data.eq_site_limit) 2.143983 seconds
1. Precompilation time: Only once when you install or upgrade a package.
2. First running time (that includes compilation time): When you call a function like `plot` the first time in a session
3. Second and subsequent running times: The compiled code is now in the cache, and what you see is the actual running time.
Thus what you are reporting here as precompilation (1) is not something that a user will see every time. What they will see is (2) for the first time in a session and (3) on subsequent plots. Could you confirm this is what you see - to ensure that those following along understand the terminology?
Here are the usage times I see when I use Plots on a daily basis in a Julia session:
~ exec '/Users/viral/Desktop/Julia_Releases/Julia-1.5.app/Contents/Resources/julia/bin/julia'
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.5.2 (2020-09-23)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release julia> @time using Plots
11.341632 seconds (20.21 M allocations: 1.108 GiB, 2.87% gc time)
julia> @time plot(1:10,1:10)
2.291147 seconds (3.24 M allocations: 165.569 MiB, 7.80% gc time)
(Another 2 seconds for the window to be drawn the first time on mac)
julia> @time plot(1:10,1:10)
0.000763 seconds (2.80 k allocations: 166.023 KiB)I am a Python data science user. If data gets big enough such that loading time is a bottleneck, I use parquet files instead of CSV, and PyArrow to load them into pandas. It’s a one line change. The creator of Pandas is now leading the Arrow project. It’s very seamless. Don’t know if I’m typical but that’s me.
Jacob Quinn (karbacca) also has a Julia package for integrating Julia into the Arrow ecosystem: https://github.com/JuliaData/Arrow.jl
For 3) there inherently is no optimal solution to "a problem with large performance-correctness tradeoffs", additionally users do not care only about speed, for a huge number of applications, absolute reproduction of precision may be the difference between a good simulation and data science quackery. We saw elsewhere in the thread that Julia's float parsing code at least presently handles certain syntaxes incorrectly. Ignoring performance, this is criteria for deciding whether a solution is fit for purpose at all.
The overall point otherwise makes sense: most DS folk don't and maybe shouldn't care about this stuff, however this is HN, and elsewhere in the comments there are plenty of examples of people who do.
You seem to be under the misapprehension that mainstream CSV libraries parse float values with reduced precision in order to go faster. They do not. A decimal floating-point string represents a precise mathematical value and the only acceptable value for a CSV parser to produce is that value correctly rounded to a Float64 (or some other type if requested). Any CSV parser that does anything else is simply incorrect.
> We saw elsewhere in the thread that Julia's float parsing code at least presently handles certain syntaxes incorrectly.
You may be referring to a bug mentioned elsewhere that fields like `-123.` with a leading minus sign and a trailing decimal point are misinterpreted as strings. That is a bug, plain and simple. It has nothing to do with float parsing speed/accuracy tradeoffs.
Comparing anything against the built-in CSV reader in Python is pretty much an exercise in nothing.
In 20 years of programming Python professionally, I have used the built in CSV parser about 5 total times, and all were use cases where 10-20x improvement of CSV parser performance did not matter whatsoever to the application.
One person may install something that’s blazingly fast. Someone else may install something because it can convert the data to an in memory parquet format, someone else may use something that can virtualize SQL queries over top of disk-backed csvs.
They should all be different for the nuance of each application.
The fact that you write these as isolated special implementations is a good thing and not a deficiency of generics.
This way it can be gradual. You only target certain functions or modules for optimized C implementation, you don’t waste static typing overhead or compliance to a generic interface on the 99% of the code that will have no material gain from any optimization.
It’s similar with gradual typing in Python as well. Annotate what you need, omit annotations for what you don’t. Implement an extension module or multiple dispatch for what you need, use plain Python for everything else.
Python also lets you easily achieve this in many ways - notably fused typing in Cython.
With numba it can be even more dramatic. Zero code changes, solely type annotation in the decorator, and you can often get 1000x speedup over pure Python, because numba can optimize away many of the extra CPython data model features that tight loop code may occasionally not need to use.
Same with Numba, despite its promises, I've never seen anything close to 100x speedups in the real world, and I've even seen a fair few cases where numba resulted in slower code compared to 'pure' python/numpy (although admittedly that was 2-3 years ago)
Sure I could probably design some pathological code where you might see 100x speed up by just adding type annotations, but that would probably require starting with some really inefficient python code.
Thankfully you don’t have to rely on me - tons of name brand Python scientific tools use Cython for speedups.
Pandas, for example, uses Cython heavily. Here is an example tutorial that may help you,
https://pandas.pydata.org/pandas-docs/stable/user_guide/enha...
That sounds like a poorly designed library to me. Instead of doing one thing well, it forces users to consume something meant as a jack of all trades, which may not meet their use case.
I wish such a thing wasn’t forced in a standard package - rather split into different ones. Then I can easily compose them according to my needs, instead of having the stdlib force feed me some set of tradeoffs I might not want.
I’m sure these extra third party libraries exist or will exist in Julia just as they do in Python and users will have lots of good options - but what a shame for people to take a needlessly myopic view that core libraries need to be obsessed either with performance or with breadth of use case coverage coupled into one library, which seems to purely be for macho bragging rights or something?
At a higher level, this seems like a classic Julia versus Python dynamic. In the Python ecosystem there are a dozen different versions of each thing that make different trade-offs and specialize in different ways. When you're picking one, you have to spend a week evaluating all the possible CSV parsers (for example).
In the Julia ecosystem, the ideal it to have one really high-quality, fast, flexible implementation that everyone collaborates on to make it great for all kinds of use cases. And the language design and the compiler make this possible, since you can write pretty abstract code and compose it with various types and get a really fast implementation. The annoying and tricky logic of parsing the CSV format only needs to be done once, and reused for all different situations.
This isn't done for "macho bragging rights" but because it is a waste of time and effort to implement a dozen half-baked versions of the same thing. When there are a dozen separate implementations of something like this, they also end up disagreeing on their behavior, which causes no end of headaches.
It saddens me to read this weird statement. It saddens me as I was a rather early adopter in Julia (v0.3.2, if I remember correctly) and really believed its mission. I don't quite understand why various ecosystems cannot peacefully coexist without these untrue and completely avoidable statements.
Yes, absolutely. Very high level design goals are completely different between those use cases and you would be forced to make mutually exclusive trade offs to solve any one of those problems - which will directly lead to concessions in terms of the other use cases. No one generic csv parser can be optimal to all three.
Design for general case speed is directly at odds with designing either for executing SQL over top of the file (in which case the trade offs are about indexing the data for record-wise or column-wise operations) or type marshaling for an extremely narrow use case like reading into Spark.
For example, parsing for speed vs parsing for Spark interop are very likely to handle datetime parsing or null parsing significantly differently.
I know it sounds too good to be true, but my limited experience so far is that it really is true, with surprisingly few pitfalls (mostly, you have to think about type instability, which will be new to most, but is really not that hard to avoid).
I’d also point out that with fused typing in Cython, it’s also trivially easy to get overhead-free multiple dispatch in Python too, and unlike Julia, this had the benefit that it can be a gradual trade off, where you only bother writing the Cython module with autogenerated multiple dispatch implementations based on different types for the ~1% of use cases where it has any impact on the application, and you get the benefit of super easy to develop, flexible Python for everything else, with none of the code liability that comes from describing multiple dispatch generics in the 99% of the code that does not benefit from it.
Julia doesn't have static typing and there are a very large set of problems that are trivially solved with it's type system.
See the tables.jl ecosystem for example
The fact that input types are dynamic and resolved at runtime (which works identically in both Julia and Python using a Cython extension module) does not mean the multiple dispatch “is dynamic” (it’s still based on a registry of types that determine which overloaded implementation to select).
The only trade-off is whether you want to be able to extend this registry of static types mapping to implementations on the fly (similar to type classes in Haskell) which Julia supports natively and Python supports via tools like numba, or you need to ahead-of-time compile it (Cython).
This is a trade-off though, between AOT resolver speed vs JIT flexibility. It’s not definitely better one way or the other, and Cython gives you a level of control over explicit language features to enable or disable (eg Exception disabling) that is much better for some use cases.
Julia can do both static and dynamic dispatch. The latter works with function barriers works to maintain codebases that are impossible to achieve with just one or the other.
There's no resolving trade-off because when Julia knows types and call time (or a union of them ) it can inline function calls.
Anyway, neither numba nor cython has the combin of features that allows for one to define a fast custom abstract array and have another package seamlessly define a subtype that works fast and with other data structures, overloading only the necessary functions. Then a third package can come and build on top using traits from the first and some weird solver.
How about multiple dispatch with dynamic (runtime) typing?
> “ switching from C to python and then back to C would ruin performance.”
This is explicitly wrong in regards to Python. Optimizing Python often involves writing an implementation directly in C (or better, Cython) and autogenerating bindings that marshal between CPython native C types and the types of your extension module. This is extremely fast - as fast as calling native C code, there is no performance penalty for this.
> A typical user will rarely ever use the built in CSV library in Python
as a criticism of the benchmark, but if a typical user does use stdlib’s csv after all, then it seems like your disagreement is perhaps with GP and not with Stefan.
If you've got a Python code base it would cost far more in terms of effort and dollars to convert it all to Julia than to just find a better way to load data. If you've got a green field project and CSV reading is a big portion of the task load then it makes sense to pick Julia I guess.
1. Python’s std csv parser is very fast & versatile, perfectly good for a range of use cases that don’t have unusual needs.
2. For any production use cases whose needs aren’t met by the std csv parser, nobody would use it and nobody has to - plenty of other options exist (rightfully) as separate packages that allow the user to choose a different point of trade-off properties if they don’t like the std module’s defaults.
Nothing says the std isn’t good enough for production - it is. Just most people will use something application-specific and they have tons of options.
Use a binary data format or hexfloats instead. Decimal floats are a pain and especially pointless if they are being generated from and converted back to IEEE 754.
Well, in Python typically one would use pandas "to parse a CSV file into a data frame", not the standard csv parser (which doesn't deal with data frames anyway), no?
Regarding your specific points:
1. Yes, it's possible to rig up some way of parsing in parallel in Python, like splitting the CSV file into multiple files and then parsing them in separate processes and then joining that. But no matter how you do it, you have to jump through extra hoops to do it. And that is not something most users do.
2. I don't entirely follow this point. Perhaps using PyArrow's parser would be faster than what is timed here, but is that what the typical Python data science user would do? Most Python users use Pandas and the CSV parsing that it provides. If that changes, then it would be good to do a new benchmark.
3. Yes, floating-point decoding is hard, but Julia's is fully accurate, so if it's also faster than Python's how is it an unfair comparison? If Julia was cheating by doing imprecise float parsing, sure, that would be a valid complaint, but it's not.
These "conflations" sound suspiciously like excuses. If float parsing is slow, make it faster. If concrete object decoding is slow, stop doing that. If threading would be faster, bite the bullet and use threading. If the language doesn't provide good support for these things, then maybe the language is at fault.