Chris' whole point is that the biggest benefits of switching to Julia will be felt by the folks that are developing the packages and libraries for others to use. He's advocating that the best way to get you to want to switch isn't incremental language-level features, but rather it's first-in-class domain-specific packages.
This is something that will take time, but Julia's language-level features are uniquely positioned to enable the development of such packages. Chris is extremely productive (and definitely an outlier), but in less than two years he managed to coordinate and build a first-in-class ecosystem for differential equations.
So he's not advocating for you to switch at all — he's advocating for folks to build the packages (like his) that will get you to want to switch.
> But for end users throwing together a 100 line script for a data analysis? I don't think that this crowd will actually see as much of a difference between other scripting languages if the packages in the other languages they are using are sufficiently performant (this isn't always true, but let's assume it is). To people who aren't "pros" in the language, it will probably look like it just has a different syntax. It will be a little faster than vectorized code in other languages if code is in type-stable functions, but most of the differences a user will notice will come from the mixture of features and performance of packages. Because of this, I am not sure if marketing the features of the language is actually the best way to approach the general audience. The general audience will be convinced Julia is worthwhile only by the package offering.
(For context, this was after several hundred words of compelling, detailed examples of Julia's language-level advantages).
EDIT: the author then goes on to claim
> that "lack of packages" isn't really a problem with the ecosystem: you can find a great package that does what you're looking for.
with several examples, pointing to poor discoverability, uniqueness, distribution and branding.
He also recommends more practical, use-case-specific how-to guides.
More importantly: I can imagine writing algorithmic improvements to it for my use case, which is where I hit a wall with python + scipy/numpy/numba.
I can't go and use my python skill to teach a scipy routine about some feature I need.
> but rather it's first-in-class domain-specific packages.
You'll note I didn't cite lack of libraries as a reason I'm reluctant to use Julia. I agree that a solid ecosystem is critical for adoption, but my point is that, for me, it's not sufficient to overcome the human/time/cost factors.
Stuff like the ODE library that Chris has, or JuMP (for mathematical programming) that Miles Lubin, Iain Dunning and Joey Huechette wrote, or a number of other packages are simply not available elsewhere. The Celeste project, for example, achieved 1.6 PetaFlop/sec of compute rate on half a million cores.
Programming languages are abstractions. Some abstractions are better for certain kind of problems. Many of the amazing libraries that are available in Julia would simply not be easy to do otherwise.
Think about it - if these were incremental advances and no serious community would ever form - then Chris would have never written his packages. Julia has seen over a million downloads. In my biased opinion, it is more than an incremental advance. I think the right approach is to understand where those advances are and apply it to the right kind of problems.
There is a balance to be struck between the two extremes of nothing new will ever get adopted, and everything should adopt anything new and shiny that comes out.
[1] https://arxiv.org/pdf/1508.01982.pdf
[2] http://jump.readthedocs.io/en/latest/installation.html#getti...
With all due respect, this is just false. I can construct a million dimensional linear program for a large network problem in a few seconds in python. Solving it with cplex can take minutes.
I have a Ph.D. in computational mathematics and do this for a living. I write lots of C, and still almost exclusively use Python or Lua for problem formulation because the performance just doesn't matter.
I'm certain there are some cases where it's true. One I've encountered is doing real-time solution of MILP for adaptive path planning. Here the problems are of moderate size and you want to reconstruct them from geometry data every 10ms or so. Here Python will indeed bite you in the problem formulation stage.
> As long as our hypothetical Blub programmer is looking down the power continuum, he knows he's looking down. Languages less powerful than Blub are obviously less powerful, because they're missing some feature he's used to. But when our hypothetical Blub programmer looks in the other direction, up the power continuum, he doesn't realize he's looking up. What he sees are merely weird languages. He probably considers them about equivalent in power to Blub, but with all this other hairy stuff thrown in as well. Blub is good enough for him, because he thinks in Blub.
Julia has offset array indexing (for essentially no additional cost) which is an amazingly useful feature! See eg. [2] Think of this feature as blurring the distinction between data (accessing an array) and computation (calling a function). The fact is that arrays as they are used (contiguous memory location collection) often carry more information than being just a dumb list of arbitrary data, and it's very convenient to expose that in their interface.
[1]: http://www.paulgraham.com/avg.html [2]: https://julialang.org/blog/2017/04/offset-arrays
As in, I don't see how its possible without stretching the meaning of "no additional cost". I'm familiar with displaced arrays and the like in Common Lisp, so i get that you can do offsets and things without using much additional memory, and i feel its a small jump from that to arbitrary indexing.
I mean, sure, if you're programming with constants or common patterns or things like that, a compiler could theoretically figure out and adjust the references itself. Or you can use macrology (I presume, since julia is so close to being lisp in matlab clothing, there's no issues with this in theory). Or you can come up with some kind of functional wrapper to adjust offset indexes back to the base case...
But if you're actually using run-time calculations using returned numbers from a calculation to index N-quadrillion times into various arrays and you're swapping between a standard and arbitrary/offset index measure, which is about the only time I can see you'd actually care about the relative efficiency of repeated arbitrary indexing, then it can't be a pre-compiled thing. The offsets between the actual numbers you're getting back and the actual index into the arrays have to be calculated in an additional step at run time.
Now its true that's likely to be only an addition or multiplication operation for each one, (and if it isn't, the run time application of another transformation just cements my point) but perhaps my notion of "essentially no additional cost" is more strict...
julia> using BenchmarkTools
using OffsetArrays
A = rand(1000)
O = OffsetArray(A, 0:999)
function naive_sum(A)
s = zero(eltype(A))
@unsafe for i in eachindex(A) # @unsafe will eventually be folded into @inbounds
s += A[i]
end
return s
end
naive_sum (generic function with 1 method)
julia> @btime naive_sum($A)
1.025 μs (0 allocations: 0 bytes)
514.0960505118594
julia> @btime naive_sum($O)
1.023 μs (0 allocations: 0 bytes)
514.0960505118594
loop:
vaddsd (%rcx), %xmm0, %xmm0
addq $8, %rcx
addq $-1, %rax
jne loop
Now there is the awesome FFTViews.jl package which goes one better with wrap-around indexing: https://github.com/JuliaArrays/FFTViews.jl
I'm skeptical any GC'd language can approach C/Rust unless they have explicit mechanisms to do data layout for using the cache/prefetcher to the fullest degree.
Julia is designed to make it extremely easy to write high performance mathematical programs with ease. Rust is designed to make it easy to do systems programming. Both are high performance for the things they are designed to do. Just like I wouldn't do systems programming in Julia (although I am certain KenoFischer would), I wouldn't want to do mathematical computing in Rust.
C/Rust/C++ fit that very well so I wanted to understand how similar claims were made for a GC'd language. C# for instance can work with value types and was wondering is Julia has similar constructs.
(As an aside, we also take this further with abstractions of 3x3 rotation matrices using a variety of internal representations such as quaternions in Rotations.jl, and allow composing coordinate transformations in CoordianteTransformations.jl, etc, etc).
Touching on the original authors post about the package ecosystem, I feel it can't be understated enough how easily different packages can compose and be used together in Julia. I like to say there is a product rule here where functionality/productivity grows as the square of the number of packages/features you can use together naturally (regardless of programming language). We can use static arrays, rotations, coordinate transformations, etc in combination with differential equation packages to simulate a system or with an optimization package to optimize a coordinate transformation for SLAM given certain measurements, and so-on, at full run-time speed but comparatively little user effort.
This is the niche that Julia wants to occupy.
To be clear, I have no beef with Julia, I'm sure it's a fantastic language. I take issue with people thinking they can get the same level of performance without explicitly controlling their memory access patterns/allocations.
98% of developers will never need it in their careers but when you do there is no substitute.
Take the JVM, a good majority of ORM databases uses Sun.misc.unsafe to do manual native memory allocation. Not because it's faster(because it isn't faster than bumping the nursery pointer).
They do it so they can control where disparate datatypes live in memory so that as a cache line is read in the prefetcher is already pulling in the next cache line.
Things like this will get you a 10-50x performance increase, which in some cases you absolutely need it.
Pauses is indeed a problem, which then requires you to manually tune the gc to your settings.
> reference counting takes more memory and slower than a well implemented gc
This is a ridiculous cliche at this point. It might be true if every memory allocation was reference counted, but in C++ (and julia) almost everything winds up on the stack. What doesn't wind of up on the stack is usually being dealt with using ownership and move semantics. The number of reference counted variables in my C++ programs is usually 0 unless they are being shared across threads. Not only that, but within a thread, move semantics means that the reference count doesn't need to be touched.
While there may be 'conventional wisdom' I have implemented non-trivial software in both C++11 and Julia, and optimizing memory allocations happens in both and Julia required more thought. Julia made data structures and general functions easier to write so it wasn't as if it was a net negative, but memory allocation wise I don't feel the garbage collector made things any easier.
Then on top of that you have the myth of the 'well implemented GC'. Java, C#, D, Go and Julia are all languages where this seems to be a constant struggle. After significant R+D some are there, but if you look at D and Julia, it is a constant user complaint.
As for cyclic data structures, I'm not sure why I would do that in the first place and I'm doubly unsure why I would do it with pointers and fragmented memory allocations.
Looking at the performance docs[2] this is pretty clear in that array types(which looks like how Julia does Matrices) get allocated on the heap. You can clearly see the performance impact in the docs(1.95s vs 0.08s) that this type of behavior has.
While you can preallocate(which the docs suggest and is the only path for GC'd languages) it's not an ideal solution. If your type is smaller than a single cache line, like in the example above, you've just flushed a whole cache line just to bring in that one pre-allocated value. You also run into the issue if you don't know exactly how many values you need upfront which leads to pooling. In that scenario your pool may be large enough that you're bouncing between cache lines on different pooled objects.
This is the type of thing that you really need control over if you want performance "as good as C". Anything less will be a compromise. There's also the whole class of zero-cost abstractions that can get from C++/Rust which leverage all the above to great effect. That lets you get things like nom[3] which gives you high level semantics + productivity while maintaining parity with C.
None of this is academic, these are all optimizations I've used on shipping products that went out to millions of users. In each case we had fixed hardware with a limited execution budget and the 5-20x improvements we made were critical to us shipping a product that people wanted to use.
[1] https://discourse.julialang.org/t/how-to-know-if-object-memo...
[2] https://docs.julialang.org/en/release-0.4/manual/performance...
[3] https://github.com/Geal/nom_benchmarks/tree/master/http / https://github.com/Geal/nom
One thing that was really common for us to do was to instance a weighted graph(something like this[1]) per-actor. This means that you might have 10-300 floating point values in a block indexed by the node they interact with. It was really common to see one, maybe two values change on a per-frame basis. With the constraint above I'm now copying 300 floating point values every time any node changes which would be brutal for performance. Or I'd take a potential cache miss each time I touched the array which could be even worse if it was a reference type.
To be clear, I'm not saying Julia isn't really good at what it does. My complaint stems from the fact that you can't claim performance good as/better than C without having all these tools at your disposal.
We haven't even gotten into things like restrict[2] where things like Rust's ownership model let you get it for free[3].
[1] https://docs.unrealengine.com/latest/INT/Engine/Rendering/Ma...
They're not. The semantics of value types and reference types are different in the presence of mutation. So if you want uniform object semantics in a language, then objects that can be implemented as values must be immutable. There are many languages that have kept these independent and bifurcated their type system instead (Java, C#), but it's been a source a great deal of pain and frustration (e.g. Java's `int` versus `Integer` awkwardness).
Fortunately, there's nothing you can do with mutation that you can't do just as well by modifying and replacing an immutable value in a mutable cell – the compiler implements them the same way. Wrap an immutable in a mutable `Ref` and voila, you've got something equivalent to a mutable value type without exposing completely incompatible semantics in the language.
I think C#'s overloaded term for "value-types" may have caused you to misinterpret the above. Let me be more clear, if a value lives on the stack or the heap is completely separable from if it is mutable or not, that's my objection.
> ... just as well by modifying and replacing an immutable value in a mutable cell
In terms of "correctness", sure. In terms of performance see my 300+ float block example above. I shouldn't have to copy, modify, copy when I can just mutate in-place(and be explicit about that rather than relying on language semantics).
Also I can seem to find any mention in the docs about Ref's semantics aside from a passing mention in the FFI section.
In my case, julia/llvm was smart enough to figure out that the read and write can be eliminated. Hence, the julia code that replaces an immutable with a copy where only few fields are changed generates the same @code_native as the obvious evil construction (figure out where the field is stored; unsafe_store! to the pointer).
But I guess this optimization is unreliable, or at least it is not well documented when this optimization is guaranteed to happen. So the situation is not optimal, but also not as catastrophic as you would have guessed without reading the generated native code.
Regarding preallocation, you tend to want to avoid dynamic memory allocation in high performance applications anyway, so whether you do that in C++ or in Julia, doesn't really make too much of a difference. It is true this is a little harder to control in GC'ed languages than in languages where you have to do memory management manually, but the effect is about the same. Julia provides tools to figure out where you're using dynamic memory allocation and those tools will certainly improve in the future.
One thing I think is under appreciated from the performance perspective though is how easily julia lets you express data layout transformation to take better advantage of the cache hierarchy. I touched on this a bit in my JuliaCon presentation [1].
Other languages such as Java have that optimization too. But it is quite limited because it can only be applied to data whose size is known at compile-time. There is also the problem that if some data allocated in a procedure is passed to another procedure, it can't be allocated on the stack because it can escape.
A programmer working in a low-level language can be smart and choose the best locations (stack or heap) for each piece of data. A language that does not allow explicit stack access (like C does), can never make as optimal decisions.
Yes! Which is why it shouldn't take me digging through 3 different documents and still getting it partly wrong!
C/C++/Rust make this easy by annotating the type modifiers(&/*/Box/etc) with how the object lives in my runtime system. Rust even gets bonus points for giving me aliasing information as well(&mut vs &).
Sure would.
You can also easily verify this yourself by looking at assembly code dumps for JITed Julia functions. You can see that the amount of code would be comparable to that of C for many common scenarios.
As for cache and data layout. That GC'd language can't control data layout and cache usage is simply wrong. Best known example is probably Go. You can nest structs to create contiguous blocks of memory in Go, just like in C. You can also create arrays of structs as contiguous blocks of memory.
Just because Java and C# doesn't work that way, doesn't mean all GC'd languages are like that.
A struct of concrete types in Julia is just like a struct in C in the way it is packed in memory. An array of structs in Julia or a nesting of structs will have similar memory layout as in C.
So I know this sounds really odd, but Julia despite being a dynamic language thus allows more optimal memory layout than a statically typed languages such as Java.
What is frustrating with promoting Julia, is that it simply sounds way too good to be true, so people dismiss the language before they have actually read up on it. I advice you to read up a bit more on the details of how the language works. It will then become clear why it can pull off stuff which seems far too good to be true.
C is the leftmost dot, Julia is just to the right.
1. Yes, some languages do not make it easy to install and link against a well-tuned BLAS. The benchmarks were very recently updated, and you can see the level of effort required to get everything installed and working properly. This was all done in the open on the discourse message board.
2. Yes, the benchmarks involve testing very specific language-level features like iteration, recursion, and IO. And they're testing the ability of specific languages to do these things. Yes, of course in Python you'd look towards a vectorized library call instead of a `for` loop, but then you're not testing Python itself anymore.
[1] https://alioth.debian.org/forum/message.php?msg_id=182495&gr...
[2] http://benchmarksgame.alioth.debian.org/play.html#languagex
[3] https://github.com/JuliaLang/julia/tree/master/test/perf/sho...
Like this guy did -- https://pybenchmarks.org/
If these benchmarks have resulted in all the languages focusing on a fast BLAS, I think that is fantastic.
- libraries
- community
- tools
Are much more important.
Take a look at the link below, comparing a simple sum in different languages (among them Python and Numpy). The power of Julia is that there is no privileged code. Your code will be as fast as the base library.
http://nbviewer.jupyter.org/github/alanedelman/18.337_2017/b...
What's liberating is that for the things which aren't covered by such packages, there's no huge penalty for just writing it yourself. In speed but in effort too -- it's quicker to write the loop than to figure out how to make np.einsum do what you have in mind.
The C function is of course naive. I'm pretty sure that a hand-unrolled loop would be faster.
Here's the standard pitch for Julia: C-like performance with R/Python/MATLAB-like expressivity.
This is undeniably true. For someone coming from R/Python/MATLAB, you will be able to program as easily in Julia, if not more easily and more cleanly, and get much better speed.
However, what I've come to realize now is that you can get something similar from Kotlin, Nim, and Crystal, but those are much more general-purpose, and often are more consistent in their performance. They don't have the same numerical libraries at the moment, but they have a much bigger use-case target, and for something like Kotlin, a lot more leverage from the community.
Rust is a lot lower-level, but as the numerical resources grow (and they inevitably will) and layers of abstraction are added, what you will probably get is something very similar to Julia in expressivity but more extensible downward, and probably more performant. Rust has a ton of room to grow, and has many good role models from C (e.g., Eigen).
This also isn't even touching on PyPy and things of that sort.
I like Julia and still reach for it when I'm starting a project. Often it's worked great, but other times I have to go back to R or Python for some library that calls C, because the Julia library, while usable in a way that pure R or Python isn't, is still not actually as fast as C. Sometimes "on the order of magnitude of C" is not actually good enough, and you need C. Also, if you poke around for benchmarks that aren't posted by the Julia developers themselves, you'll see different conclusions about the real-world performance of Julia, so there's that.
I guess I read articles like the linked post, and I end up feeling like they're misleading. Is Julia better for data analysis than R or Python? Yes, almost certainly if you were starting from scratch and not calling any products of outside languages. Is Julia better than R or Python + C as it is in the real world? Not sure, because you still have to call C anyway at some point if you're being honest. Is it better than Kotlin, Nim, or Rust? I'm even less sure. If Julia wants to maintain long-term competitiveness, it has to reach out in the opposite direction of those languages, from numerics -> general purpose computing, because they are inevitably moving from general purpose computing -> numerics, and they are bringing a lot with them.
e.g. Kotlin is being used for Android apps, which is worlds away from what MATLAB and R users do.
This view of yours as applied to something new is as old as the woods, progress and shifts take place regardless of that. Small mercies actually but some are still writing and maintaining COBOL applications and earning extremely respectable salaries from it.
All you are really saying is you will not be an early adopter. That's OK, there will be others. Your non-adoption is unlikely to be a shattering loss for you or for Julia.
Regarding whether improvements are minor or significant that's a matter of opinion.
Julia has great interoperability between Python (checkout PyCall.jl) and many other languages. You don't need to abandon your old codebase.
With regards to your other points, a lot of that is subjective and hard to argue about, but for me Julia has drastically changed the way I approach writing scientific code and is not just an incremental improvement upon the status quo.
(assuming we're not just arguing over arbitrary-array indexing to input predefined constants at the REPL/code level).
To access an array via index, you basically have to take the reference to the array, the index (i), and generally multiply (i) by the space set aside for each element (e).
i * e
If you've offset the array, you have to take the offset index (o), and call a mapping function (m) that maps (o) to (i), so the result can be multiplied by (e). Though it should be said there is another theoretical alternative: you can figure out a way to map (o) directly to (i * e) without the intermediate operation or mapping (o) to (i).
Where (i) and (o), and the relationship between them, is known at compile time, you can theoretically get the compiler to magic that operation away.
But where (o) is only known at run time, you have to first fetch/generate (o), apply (m) to map it back to (i), then multiply (i) by (e) before knowing how to access an array.
If you know fetch/generate (i) at run-time, you don't need the map operation (m), you just multiple (i) by (e) and there's your array reference. But otherwise there's an operation (m) sitting around whose cost must be paid.
You can't handwave it away via reliance on cpu parallel instructions or anything like that because its by definition a serial operation: you can't multiply by (e) until after you've got (i), you can't get (i) until you've applied (m), and you can't apply (m) until after you've fetched (o).
Now where the cost of (m) is sufficiently low relative to other operations, or where the number of array accesses are low, the two may appear sufficiently similar and offset array access might appear practically costless. But the point where we generally care about these things is because we're doing them a very very large number of times on a sufficiently large number of indexes that can't be known or optimised away ahead of time.
And it is worth pointing out that addition operations and things are remarkably cheap in general. But if you're accessing these arrays themselves in order to do addition operations or something similar on their contents, then it may still take up a relatively high proportional amount of time relative to the entirety of the program/operation.
Avoiding theoretical discussion on the possibility of functions (m) that map (o) directly to (i * e) while being equal or cheaper than (i * e) in cost, there is another possibility that might appear in practice...
If the original array access operations are not optimal, (or slowed down due to other issues, such as in-optimal caching or memory access, then it may be possible for both offsets and array access to be equal in practice. If indexing by both (i) and indexing by (o) are both seeing an additional step between the getting of the index and multiplying by element size, or if there is sufficient lag because the CPU isn't doing work optimally, then in profiling and practice, both would appear to be running to the same speed. But this wouldn't imply you've optimised offset indexes, it implies your regular indexes and operations aren'y optimised enough already....that's why it appears you can get something for free.
The one exception to that is when switching between languages. Sometimes one forgets which language one is in and a syntax error will result. Like using the dot operator to concatenate strings in Python or leaving semicolons off the end of PHP statements.
Note: Although I'd side with EWD on this one (for now, at least), I don't intend to reference it as an "appeal to authority", but as an example of someone producing a reasonable, properly worded and thought-through argument on the topic, which I've not really seen since. I'd be very interested in what similar arguments would look like from the other perspective. If someone would like to convince me, feel free give me a link to some reasoning :)
Also: I personally find "it's always been like this" to be a terrible argument in either case. Those are the reason we weren't able to standardize for one system of units, for example. In the Swiss army, there's a saying that is used to explain the various idiotic decisions and the bogus explications thereof a soldier has to endure:
Ist so weil ist so, bleibt so weil war so.
Things are like that because they are like that.
All stays that way because it was that way.
I have my opinion, and that is that zero is the proper number to start enumerations on, not just in programming, but for everything.
Everyone starts counting objects at 1. Zero means an absence of objects to count or list. I count that there are 7 cars in the lot. The first one is 1. If there were 0 cars, I would not have any to enumerate over. If I make a list, I always number the first one as 1 (or A). If there was nothing on my list, I would have nothing (0) to number.
Word processors start list numbering at 1. Printers number the first page as 1. Kids are taught to count starting with one, using one finger. I feel like this is only an argument because we have zero-based indexing in mainstream programming languages, although maybe there are some mathematicians who think otherwise?
I don't take programming languages as a reference point. It's the other way around. I try to figure out what the most natural, principled, and elegant way would be, and I would base my programming language on that, if I were to design one.
> Zero means an absence of objects to count or list. I count that there are 7 cars in the lot. The first one is 1. If there were 0 cars, I would not have any to enumerate over. If I make a list, I always number the first one as 1 (or A). If there was nothing on my list, I would have nothing (0) to number.
We should distinguish between two uses of the natural numbers, I think:
0. _Ordinal numbers_, for assigning labels to things. "This is car 3." 1. _Cardinal numbers_, for saying how many things there are. "There are 7 cars."
The relationship between the two would be that the number of cars is the number that is next in line when you have called out the indices (0, 1, 2, 3, 4, 5, 6) of all the cars.
> Word processors start list numbering at 1. Printers number the first page as 1. Kids are taught to count starting with one, using one finger. I feel like this is only an argument because we have zero-based indexing in mainstream programming languages
What is the most common isn't what determines, for me, what is the most proper. I will play along and use 1-based indexing when I have too, but in my heart I know how I think it should _really_ be.
> maybe there are some mathematicians who think otherwise?
Yes, though starting numberings on 1 is still common (maybe the most common). Zero is at least recognised as a natural number nowadays. Historically, this wasn't the case. Apparently, Peano started on 1 in his axioms for the natural numbers [0], and zero, as far as I understand, hardly had the status of being a number at all during much of Greek and Roman antiquity [1].
[0] https://en.wikipedia.org/wiki/Peano_axioms#Formulation [1] https://en.wikipedia.org/wiki/0#Classical_antiquity
But who thinks that way? I just see 7 cars, not an indexed listing. And anyway, the 0 indexed car doesn't exist.
The question for me is what one _should_ do, not what is usually done.