A language doesn't necessarily have to give all the old programmers an incentive to switch, if it can position itself as a good language for new programmers to learn.
For example: at our institute (computational biology), we had a PhD student who was an early Julia adopter and wrote his model in that. Several students have since joined the project he started, so obviously they're now writing Julia too. That project's experiences with the language were so good, it soon became obvious that for our use case, Julia was superior to any other language we'd used so far. So pretty much the whole research group has now shifted to Julia, and that's what we teach new students. Slowly, other groups in our institute became interested, and more and more people are adopting it, which in turn means that their new students will also end up learning it in future.
Being able to iterate and mangle huge columns with real lambdas and without having to marshal arguments to/from C++ is a huge advantage.
Where I used to spend hours in aggregate searching through docs for pandas/numpy, for stupid shit like "how do I shift but also skip NaNs", now I just write a for-loop in a couple minutes and get on with my work.
There's a whole subclass of tasks in R/pandas to work around the interpreter that just aren't needed in Julia.
For me at least it's well worth the syntactical warts and slow interpreter.
The truly huge advantage for Julia is how it plays with parralelism. The GIL makes it an absolute pain to do parallelism in python. Always ends up in threading hacks with numba or joblib, or multiprocessing, which has its own unfixable flaws
Although Julia is a growing alternative to Fortran/C/etc for long-running computations, it remains awkward and unpleasant for interactive analysis. Users familiar with Python/R/etc must weight the benefits of Julia against its slow library startup, its cryptic error messages, and its thin documentation.
Also, the lack of a community repository for well-vetted Julia libraries can limit uptake by professional researchers who must be able to trust their tools. A real strength of R (in comparison not just to Julia but also to python) is that such a repository exists, and that it has automated testing across a range of computer architectures and versions of R, including not just unit testing within individual libraries, but also testing of related libraries.
Did you mean to write short-running scripts? If anything, the Julia dev workflow is biased towards interactive analysis in a REPL a la R or IPython/Jupyter. I don't mean to imply that there's no startup overhead, but how often are you restarting the REPL when doing EDA? Unless it's more than once every few minutes (which is a very odd workflow), then startup overhead is effectively amortized.
> A real strength of R (in comparison not just to Julia but also to python) is that such a repository exists
CRAN is certainly a cut above many other package repos here, but I'm not sure "trust their tools" can apply to all packages on there. Anecdotally, I've had a lot of issues with compiled dependencies and missing/out of date external assets on less well-trodden packages. There's a reason MRAN, Conda and JuliaBinaryWrappers exist after all.
For whatever reason, Julia package maintainers also seem more receptive to making their work compatible with other libraries as well. This goes beyond just multiple dispatch as well--imagine if tidyverse/non-tidyverse wasn't such a hard split.
If you care about performance in code that mixes together several packages in nontrivial ways, Julia is way better than Python.
There's a far broader range of libraries in Python than Julia, but none of them are going to prevent adoption of Julia when its performance advantages are crucial, because of the excellent facilities for using Python from Julia.
It is overkill/brute force to install all the Anaconda default packages when a beginner is not going to use over maybe 5-10 libraries, but it's a solution that has worked flawlessly for beginners from my experience watching non-software engineers and "non-technical" people using Linux, Windows, and MacOS try Python for the first time in math and data science classes.
Julia is using LLVM for code-gen has to compile a lot of code before you can actually use stuff like plots.
It takes ages to get a Pluto Notebook up and running, while a jupyter notebook is available instantly.
But yeah latency kills :)
Oh, man, this is indeed a major feature. My main point of friction with jupyter notebooks is the stupid json ipynb format. Why can't it be just a regular language file with comments?
Have you ever used Jupyter notebooks? They contain code, rendered Markdown, images, plots, video players, widgets, etc.
How do you see a "regular language file with comments" supporting this, instead of the "stupid ipynb format"?
You can use plain text files with Jupyter, too.
The code could be verbatim python code (or whatever language the notebook uses), and the rest could be embedded inside comments. I don't see any problem with that (besides the very concept of "rendered Markdown" being totally out of order). The fact that they are saving it as json by default seems more to be laziness by the developers than a well thought-out solution, that could be just a straightforward serializer.
Do you mean embedding images and plots inside comments? If yes, please elaborate on how you see that happening in the real world.
>The fact that they are saving it as json by default seems more to be laziness by the developers than a well thought-out solution, that could be just a straightforward serializer.
So, how would that well thought-out solution in the form of a "straightforward serializer" work? I have a flat file, and I want to display images, plots that you can zoom into out of, figures, etc. as comments. How would that happen?
At the very least, you could put the whole json stuff inside a comment. It's already plain text, isn't it?
So instead of having the whole file as JSON, which is lazy and not well thought-out, we'll put all content in JSON, then put that JSON inside a comment in a plain text file. Do I read you correctly?
I feel we're making progress faster than these lazy Jupyter org bandits.
Only the "output" content. The code inside the cells is verbatim, and the markdwon cells are regular text comments.
See, I'm not discussing you just because. I have a legitimate problem with ipynb: very often I want to run the code of a notebook from the command line, or import it from another python program. This is quite cumbersome with the ipynb, but it would be trivial if it was a simple program with comments.
Beside this, if you only infrequently install/update package, you can use PackageCompiler.jl. I use it for PyPlot.jl (based on matplotlib), DataFrames.jl, ... and plotting some data quasi instantaneous as it is in python (even the very first time in a session).
For other areas, like web programming, there is no sign of Julia replacing Python in the forseable future.
No, it's isn't. Julia is growing but it's far from overtaking Python at this point.
> For other areas, like web programming, there is no sign of Julia replacing Python in the forseable future.
That's where Go comes in.
It's nice that Julia is getting noticed, but it's a distant blip in the radar.
The sci community is really hard to move from existing battle-tested and performant libraries.
This need to rewrite, of course, is what Julia is trying to avoid. My workflow is exactly the same, and I’d love to be able to write code in a high-level language like Python and then use that directly instead of having to rewrite.
However, in my case the reason for rewriting isn’t just performance, but also to be able to build compiled binaries. Julia aims to be as high-level as Python but faster - is there a language that’s as high-level as Python but AOT-compiled?
Common Lisp, Ocaml for example.
The “need to rewrite” is actually a sort of advantage with Cython. You only target small pieces of your program to be compiled to C or C++ for optimization, and the rest where runtime is already fast enough or otherwise doesn’t matter, you seamlessly write in plain Python.
Using extension modules is just a time-tested, highly organized, modular, robust design pattern.
Julia and others do themselves a disservice by trying to make “the whole language automatically optimized” which counter-intuitively is worse than make the language overall optimized for flexibility instead of speed, yet with an easy system to patch optimization modules anywhere they are needed.
The problem with cython is that to really get the performance benefits your code looks almost like C.
I agree with you on the optimize the bits that matter, often the performance critical parts are very small fractions of the overall code base.
I really don't get this. I'am fully on the side that limitations may increase design quality. E.g I accept the argument that Haskell immutability often leads to good design, I also believe the same true for Rust ownership rules (it often forces a design where components have a well defined responsibility: this component only manages resource X starting from { until }.)
But having a performance boundary between components, why would that help?
E.g. This algorithm will be fast with floats but will be slow with complex numbers. Or: You can provide X,Y as callback function to our component, it will be blessed and fast, but providing your custom function Z it will be slow.
So you should implement support for callback Z in a different layer but not for callback X,Y, and you should rewrite your algorithm in a lower level layer just to support complex numbers. Will this really lead to a better design?
It helps precisely so you don’t pay premature abstraction costs to over-generalize the performance patterns.
One of my biggest complaints with Julia is that zealots for the language insist these permeating abstractions are costless, but they totally aren’t. Sometimes I’m way better off if not everything up the entire language stack is differentiable and carries baggage with it needed for that underlying architecture. But Julia hasn’t given me the choice of this little piece that does benefit from it vs that little piece that, by virtue of being built on top of the same differentiability, is just bloat or premature optimization.
> “you should rewrite your algorithm in a lower level layer just to support complex numbers.”
Yes, precisely. This maximally avoids premature abstraction and premature extensibility. And if, like in Cython, the process of “rewriting” the algorithm is essentially instantaneous, easy, pleasant to work with, then the cost is even lower.
This is why you have such a spectrum in Python.
1. Create restricted computation domains (eg numpy API, pandas API, tensorflow API)
2. Allow each to pursue optimization independently, with clear boundaries and API constraints if you want to hook in
3. When possible, automate large classes of transpilation from outside the separate restricted computation domains to inside them (eg JITs like numba), but never seek a pan-everything JIT that destroys the clear boundaries
4. For everything else (eg cases where you deliberately don’t want a JIT auto-optimizing because you need to restrict the scope or you need finer control), use Cython and write your Python modules seamlessly with some optimization-targeting patches in C/C++ and the rest in just normal, easy to use Python.
This sounds like it might be interesting, but your later comments about overhead and abstraction costs sounds like you maybe don't understand what Julia's JIT is actually doing and how it leverages multiple dispatch and unboxing. Could you be a bit more concrete?
And yes, there are ways to AOT compile as well.
My anecdata kind of tells me that Go is reasonably big, but it's not yet near .NET and Java, worldwide. But it could get there in a few years, I've seen/heard about some enterprises adopting it.
Look at the whole Cloud Native Foundation thing, I think most of their projects are developed using Go.
So if you're using that stack, it's easy to assume that all new development everywhere is in Go.
It will probably balance out once the newness wears off Go (I think this is already happening).
https://deislabs.io/posts/still-rusting-one-year-later/
I should also note that after creating the initial support for Go on VSCode, they have given it away to Google to maintain it.
The issue with regards to web programming/other programming is important, because sometimes it's useful to make a website/build another tool as a scientist. Python can do both easily.
I agree. In fact if Julia hasn't overtaken Python in numerical computing by January 2022 I will consider it a huge failure.
That helps the adoption story quite a bit. You can do the number-crunching in Julia where performance counts, and then analyse and present the results using Python.
- I've never run into unsolveable performance issues with Python
So I guess I'm not in the target audience unless I just happen to be curious about a new language? That's kind of my overall point - even if Julia is a good language on its own and I work in data science, I don't have reasons to pick it over Python.
https://fluxml.ai/Flux.jl/stable/
Is still very barebones compared to Torch/TF/Flax and I would be hamstringing myself by switching to Julia even if I find the language otherwise attractive.
https://github.com/FluxML/Flux.jl/issues/1431
They are going for feature parity with pytorch hopefully in the near term.
But then times change... It’s usually the tooling and libraries. Now I don’t want to go back to Perl.
And I’d certainly be willing to give Julia a try.
A similar enabler in a new field could help Julia burst in as a general language. My 2c.
Also python is still doing great with Jax and PyTorch.
I don't know much about Jax. I've seen competent benchmarks showing an order of magnitude benefit for using ReverseDiff from the AutoDiff suite over Autograd, which is what Pytorch uses for reverse-mode autodiff
It really should be updated to have basically the content of this thread https://discourse.julialang.org/t/state-of-automatic-differe...
Python 2.0 was released in 2000. Python 1.0 was 1994, and Python 0.9 (first public release?) was 1991.
Check the google trends. https://trends.google.com/trends/explore?date=all&geo=US&q=p... Its unclear when perl peaked since it has been in decline since before 2004 But it wasn't til late 2007 that Python overtook Perl in google searches
Even while it was in decline people were still making that argument.
People like to substitute "10x better" here but I think the real number is 100,000x better, aka it's not possible by default. Q: What it would take to replace Windows? A: iPhone was a new product category that targetted a new market.
Just 6 years ago, I was taught Perl in my Introduction to Bioinformatics course. The teachers were still using Perl because it used to be the go-to language for bioinformaticians. The year after, and every year since, they've taught using Python.
Python has gotten exceptionally lucky. I am sure the two or three remaining perl users on the planet are also on HN and ready to jump to its defense, but to me this just goes to show you how heavy the switching cost is for something like this is and also how lucky python was to have been the best language to switch to at this point. It was in the right place at the right time for a lot of these switches away from older languages in obvious decline and then it was able to leverage numpy and scikit to pick up a lot of additional momentum in ML and data science tasks. It is almost never the 'best' language for the job, but coming in as second choice on most tasks is a huge win.
Jack of all trades, master of none, but oftentimes better than some are at one.
- The beginner experience in Julia is still much worse than it is in Python. Stuff that should work intuitively sometimes doesn't, and when you get a cryptic error message, it's difficult to find relevant help online. And when you do find help, some of it is out of date because the language has changed over the past few years.
- You can squeeze a lot of performance out of Python and the ecosystem of libraries is hard to beat.
- Julia has to be way better than Python to give people an incentive to switch. Being just marginally better in some aspects of the language isn't enough. And it's very difficult to be much better than Python especially in useability and ecosystem.