The existence of ad-hoc solutions like numba or cython also do not speak well of the limitations of the language. Most specialized science packages (qutip for instance, for quantum optics), reimplement tons of solver/integration/optimization logic and require plenty of cython (or C) code. Scipy does not play nice with most of the automatic differentiation libraries.
In that context (a pretty universal context), Julia is looking pretty stable (relatively) and future proof in comparison. E.g. you can not really interoperate with scipy/tensorflow in a meaningful high-performance way. The value proposition of Julia is that this will be possible, today, and hopefully when future new paradigms appear.
Also, on a tangent, Julia already has ODE solvers and Probabilistic Programming packages lightlyears ahead of what Python has.
Leaving some performance critical stuff in C is okay. We can expose them to a variety of languages then.
> away from Theano [...] I had to switch from Tensorflow 1 to Tensorflow 2
I feel you. But I think this has more to do with Google's deprecation culture. TF1 was nice for its time, but I will never adopt TF2, as they will drop it any time as well. I have higher hopes for PyTorch though.
Julia is now experimenting with pure-Julia code that is faster than many LAPACK/BLAS implementations. But I actually care little about the fact that they are a bit faster. Same with the new ODE package in Julia, which is unsurpassed in terms of functionality. The actual reason these pure-Julia packages excite me is that they are generic enough to run relatively easily on different hardware (GPUs for instance), they are on top of modern compilers (LLVM), and most importantly, they are written in a language that permits much higher level of interoperability.
To rephrase it much more simply: The fact that you can not use scipy special functions efficiently inside of any of the autodiff frameworks is exemplary of how incredibly limiting python is, when it comes to developing new numerical techniques. Julia does not have that problem as far as we know.
Only if they were specifically written for GPUs. GPUs are incredibly limited. Expecting to run arbitrary code on them is only going to disappoint you. If you could just run regular code on GPUs then everyone would have done so. Programming language support is not a barrier. The GPU itself is. In fact, you should consider yourself lucky if you have problems that don't need GPUs.
Sure, no arbitrary loops though,..but a constrained subset will work (see kernel abstractions.jl)
IMO this is worth very little; as long as there is a heavily optimized version written in C/assembly (ie MKL), the fact that Julia might be faster than standard-but-slower versions like OpenBLAS doesn't actually gain me anything as a user.
Mostly the bits about how you have to workaround the Julia GC.
So it doesn't seem like a great option for libraries that might have lots of different bindings, where C/C++/Fortran (and maybe Rust?) seem to be the only reasonable choices.
I disagree on account of the glovebox problem.
If you can't touch your data without losing perf, your API becomes awkward like a glovebox. All meaningful low-level operations must be represented in your high-level API which bloats the heck out of it. To add insult to injury, it still usually doesn't get you 100% functionality coverage, so projects that veer even slightly off the beaten path are all but guaranteed to involve C and bridge crossings and explode in complexity. So the end result is unnecessarily bloated libraries that unnecessarily constrain your creativity.
Yes, I know about numba -- I've used it heavily and even contributed a few bugfixes -- but it's often not even a workable solution, let alone a good solution, let alone a good solution that's fully integrated into the ecosystem (i.e. debug and in other people's code).
Most of the code I write is still python, but I need perf, so I suffer at the python/C bridge regularly. Like any other programming language bridge, it has trolls under it, so if you only need to cross once or twice you're probably fine but if you frequent the bridge they're going to periodically pop up and beat the shit out of you. This won't affect people following tensorflow tutorials but it greatly affects people in the deep end of the ecosystem and it really sucks.
Next time the trolls come out, I'm going to try to make the case for Julia again. Hopefully one of these times we make the right choice, but in the meantime it would be nice if the people using my libraries understood that I was taking troll-beatings on their behalf and listened to my requests that maybe we should do something about it one of these days.
New languages on the other side don't have as much baggage (and can start with increased knowledge from previous attempts) so they are able to freely potentially target better local maxima that old languages cannot anymore. Plus in this particular case, Python was not really designed to target the scientific domain (nor ML which did not even exist in the way we use nowadays), it was repeatedly retrofitted with possibly a lot more effort than would be necessary to create something better from scratch.
*Elaborated in my other comment on GP https://news.ycombinator.com/item?id=24748107
There were very real pain points with Fortran, C++, MATLAB etc. compared to Python. Now this is hard for me to argue in a few sentences as it's a distillation of having worked with many languages, but I feel that Python strikes a great balance for productivity. It feels like pouring your thoughts directly onto the screen, with no need to fight the language. The syntax is very clear, concise, "Pythonic" is a word of praise. As I said, this kind of debate is rarely productive and devolves to religious arguments, but that's my stance.
The only issue with Python is speed, but if you use Numpy properly, even that isn't a big issue.
> and can start with increased knowledge from previous attempts
They often don't make use of that out of a sense of pride and not invented here.
It ended up being a rant, but I have many pain points with Julia as well, but it's still a new language that has more space to evolve and find ways to solve them, and if the solution is yet another language that solves all of them, I'll quickly jump. I spend a lot of time programming, so any significant improvements on the usage of my time is worth the effort in learning.
NumPy should be enough for general computation. If you need autodiff or GPU then add in PyTorch. Pandas is more about various metadata than the actual numerical computing. If you want those types of features, the complexity doesn't disappear if you go to a different language.
There's an effect where a new generation of developers see complexity built by the earlier generation, say it's too complicated and mess up, we don't need all that, so start over clean and it all looks so easy. But it's deceptive, because it will get complicated again once you put in all the features but it will look familiar now to this generation of developers as they grow side-by-side with the new language/framework. After a few years the cycle repeats and a new generation says "what's all this mess, why do I need to juggle all this, I just need XY."
> Also Python for a dynamic language has pretty mediocre interactive story (using ptpython since the default repl is unusable), even simple things like copy pasting to a REPL can end up being a pain because of indenting, and the repl is far away from Lisp, Clojure and even Julia and Elixir.
Use Jupyter Notebooks (or IPython if you don't want to leave the shell).
> disappointed with Python's multithreading story
Thread pools (executors, futures etc.) and process pools (multiprocessing module) work quite nicely.
Julia doesn't even need full parity with numpy because you can trivially write your needs in straightforward Julia (in fact Julia does not have numpy, only Julia arrays). And Pytorch equivalent? Also uses Julia arrays and straightforward methods (as you can just differentiate Julia code directly). Pandas equivalent? It's a wrapper over Julia arrays. Need named tensors? Just pick another wrapper over Julia arrays and it will work on Julia's Pytorch equivalent without changing anything.
I do have to deal with Jupyter Notebooks, but they do have a lot of "real pain points" as well (it's always a pain when a data scientist gives one to add for deployment, half the time it does not work because it does not keep track of the whole execution history and need a major rewrite), but the main issue is the disconnect between the exploratory code and the production code. Julia's workflow (Revise + REPL) means I still write the structured code as usual, but every change in the code reflects in the REPL, like I'm interacting with the production code directly (and the Julia language supports a lot of commands to interface with the compiler, from finding source code to dumping the full structure of any object and even all types inferred). And of course, Julia also has Jupyter in the first place as it's name suggests, and a very cool alternative for some workflows like Pluto.jl that improves on reproducibility).
You might also want to try Erlang/Elixir, it's really amazing how you can trivially spawn hundreds of thousands of tasks that are incredibly robust even though it's a dynamic language (since you add not only tasks to do something but also to monitor and handle errors). And you can even connect and hot swap parts of the application live (as an example of superior interactivity).
I'm not saying that Python is not good, it wouldn't get where it is if it weren't. I'm saying that things can be way better, and we already have examples of it in most particular areas, but none that covers all of Python's strengths, which is why it will still be king for the foreseeable future. But I can only hope that we will get better and better tools to tackle more and more complex problems in the most simple and optimal way that we can given all that we learned as a community.
Numpy leaves a couple of orders of magnitude of performance on the table, especially if you use small arrays and a lot of intermediary computation. It is terrible in terms of memory allocation overhead. Cython or @tensorflow.function fix much of that, but then you are not really using python anymore.
> They often don't make use of that out of a sense of pride and not invented here.
This is a lazy and rather hurtful accusation.
In my case, I wrote the algorithm in Numpy/python myself which was almost unusably slow. I then outsourced it to a precompiled fortran program. Finally I just switched to Julia. There was already a library, MittagLeffler.jl which did everything I needed and was written in pure Julia. In the end everything was much faster than the Python/Fortran frankenstein code I had.
Edit: For more info on Julia and difficult (computationally) mathematical functions see:
You are handwaving away a big reason why there is research into non-Python solutions. Unless you work at FAANG, or some other institution where developer time is much, much greater than compute time; simply using a more performant solution can save either in days of waiting or thousands of dollars a month in compute.
This is literally the pain point that Julia set out to solve. Nobody uses Python for real programs, they use an ad-hoc kludge of Python and C/C++. This is basically fine if you're doing bread-and-butter things that are well covered by existing libraries, but it's a serious drag if you're trying to do anything really interesting.
Python still suits a lot of people and that's fine, but "mature" is very much a double-edged sword. We're in the middle of a scientific computing renaissance right now and Python really isn't keeping up; it was designed by and for hackers, but Julia is a multi-disciplinary effort.
What does "hackers" here even mean? Is it just a throw-away disparaging term?
Hacker has at least three distinct meanings: deeply knowledgeable software developers, shallowly knowledgeable programmers, and people who break into computer security systems.
I assume you are not using the first of these, else you would likely say the Julia developers are hackers too. And you are definitely not using the third.
van Rossum's training, for example, came through the language design and implementation process for ABC.
Tim Peters, another early designer, worked on compilers for Kendall Square Research and Cray Research supercomputers. (Eg, https://bugs.python.org/msg303574 describes developing libm for KSR, and http://stackless.com/coroutines.tim.peters.html shows his knowledge of features of Icon and Simula 67.)
So already there it was a multi-disciplinary effort.
What don't you say Python was designed for an era of single-core computing and for tasks where raw performance took second place to usability instead of going hand-in-hand? Seems rather more correct and less needlessly antagonistic.
But you seem rather fond of needless antagonisms, as people certainly do write "real" programs all in Python.
No, my point is that the Julia team includes a lot of people who aren't primarily software developers. Many of the top committers to Julia are working scientists who have first-hand experience of running HPC jobs with Julia; their input is a crucial factor in the success of Julia as a numerical computing language.
Python is a very good scripting language, but it's not built for numerical computing and it's not built by people who understand the needs of scientific users. It's not a bad tool, it's just the wrong tool for the particular jobs that Julia was built for.
In that case, your statement that Python "was designed by and for hackers" is not correct. You can go back to van Rossum's 1993 paper on Python, at https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.38.... and see:
> Python is used or proposed as an application development language and as an extension language for non-expert programmers by several commercial software vendors. It has also been used successfully for several large non-commercial software projects (one a continent away from the author’s institute). It is also in use to teach programming concepts to computer science students
"non-expert programmers" and "computer science students" are not hackers by your definition, resulting in an internal inconsistency in your claim.
Python indeed wasn't built for doing numerical computing in Python. But it was definitely influenced by the needs of people who 'have first-hand experience of running HPC jobs' - under the "steering" model.
Here's a representative abstract from 1996, quoting https://www.osti.gov/biblio/461415 :
> We have described current approaches and future plans for steering C++ application, running Python on parallel platforms, and combination of Tk interface and Python interpreter in steering computations. In addition, there has been significant enhancement in the Gist module. Tk mega widgets has been implemented for a few physics applications. We have also written Python interface to SIJLO, a data storage package used as an interface to a visualization system named MeshTv. Python is being used to control large-scale simulations (molecular dynamics in particular) running on the CM-5 and T3D at LANL as well. A few other code development projects at LLNL are either using or considering Python as their steering shells. In summary, the merits of Python have been appreciated by more and more people in the scientific computation community.
One of the authors of that presentation, Paul Dubois, was lead developer for Numerical Python, which evolved into NumPy.
The needs of numeric computing, under the steering model, contributed to Python-the-language. Most recently with "@", but also the multiple slice notation a[1:3, 5:11:2] and the change to remove the three-state compare cmp(). Plus under-the-covers support for a wide range of HPC architectures.
In turn, Python's support for steering numeric computing helped it make in-roads in HPC, where it remains to this day.
You may call the steering model 'an ad-hoc kludge of Python and C/C++', but as I've pointed out, you have a tendency to use needless antagonisms, and that's another one of them.
You're the one who claims I don't do "real" programs, only "bread-and-butter things" simply because I don't do bare-metal numeric computing software. Just how much of a multi-disciplinary effort is Julia if bare-metal numeric computing software is the only discipline it focuses on?
That's a big one. Memory usage and typing are other issues. That it doesn't really run in the browser is another. Or that it has more issues than C++ when trying to write a portable GUI.
It's fine for education and prototyping.
In other words, there is an even crankier and older community for which Julia may be the first actual change in a very long time.
This is particularly the case in HPC, where everyone still uses Fortran, C, or C++, and has never moved nor ever will move to Python because Python is fundamentally unsuited to these workloads. But in some cases, Julia is [1].
The best differential equation solvers (e.g., SUNDIALS [2] for a modern example) have been written in FORTRAN for the last 50 years. If Julia can challenge Fortran as the go-to language for this type of work (e.g. DifferentialEquations.jl [3] and SciML), that would hardly count as excessive churn.
[1] https://github.com/jeff-regier/Celeste.jl
Because underlying assumptions change.
Python still has an enormous amount of baggage from its single-threaded and fully-interpreted past (and present, really -- but I'd expect fans to argue). Should we live with the kludgey, limited workarounds? Or switch to an ecosystem built on a more modern foundation? Only time will tell -- but I won't shed any tears for python if we decide on the second option.
I also believe that it has drawn too much from MATLAB only for the sake of being familiar to that group and not the computer science and software engineering communities though such as the one based indexing you mentioned. Unfortunately that ship has already sailed and we will likely have to wait another 10 years for the next one to hopefully fix those problems.
Julia claims that it wants to solve the two language problem. To do that it must win both the scientific compute, HPC, data science, machine learning, etc. group and the CS, software engineering, dev ops group. I've shown Julia to quite a few members of the latter and they always love the idea and technical capabilities of Julia but visibly cringe when they find out it uses one based indexing. They also almost universally find the syntax ugly.
Unfortunately the available packages to change that also does not help because most people will never change the defaults and IMHO it would be bad to do so. When writing Python you follow the PEP-8 style guide, it is just best practice to follow the languages conventions so everyone is on the same page. This is the same problem with C++ trying to add all these new features and a solid package manager like Rust has, it will never fly, people just won't use it everywhere, it isn't going to be in the standard library, It isn't the default and it isn't part of the culture.
This is a sunk cost argument. Python’s stack got us a long way, but it still falls short in a lot of ways. First of all, it is eager so you can’t build up an entire query, have the system optimize it, and then execute it for greater performance. Secondly, you can’t easily parallelize operations on the same dataframe. Thirdly, it’s poorly suited to running in a web process. Fourthly, it’s all written in C/C++, so debugging, building/packaging/installation (esp on niche systems), etc is a pain and it prevents many of the optimizations Julia and others can do natively, etc. Fifthly, the APIs were horribly designed (especially matplotlib, good grief). Etc.
I don’t know about Julia in general, but in most domains there’s a middle ground where people can have nice, thoughtfully designed tools (albeit with minor issues), and endless churn. It certainly feels like the scientific toolchain is one of those areas—fix a few of these enormous, glaring problems and then enjoy some stability.
I've seen comments like this fairly often, but am not really sure what about matplotlib's API makes it so bad. Hoping to learn why.
The only equal-basis competitor to matplotlib that I'm familiar with is Matlab, which is definitely worse. Declarative plotting libraries like ggplot2 have a nicer API, but the grammar of graphics approach is an entirely different paradigm, which gets as thorny as anything else if you have to do something unexpected. For the kind of medium-level plotting where you're given prepared plot elements, but have to explicitly specify where to draw them in data or figure coords, what would be an example of a "good" API (better than matplotlib)?
To your point though, I do see a lot of unnecessary disruption particularly in the web dev world. I think people like working on new stuff. It's exciting to take the first steps, see order of magnitude improvements every commit.
Electric motors reached a technological plateau. As https://en.wikipedia.org/wiki/Electric_car points out, it required MOSFET power converters and lithium-ion batteries. Before then, it could appear that it reached a technological limit.
At what level of language improvement does it warrant rebuilding things? Eg: What if there has actually been a change as revolutionary as multiple dispatch based on a good type system, which cannot be backported to Python, and provides huge advantages for a diverse set of applications? For many users, it would be crazy not to switch... especially given that it’s actually much easier to write performance and flexible libraries in the Julia ecosystem (writing Julia code) than in the Python ecosystem (writing C/C++/Cython code).
You might be succumbing to a bit of the blub paradox here... Python might be good enough for all your needs, but people are justifiably enamored with Julia because it brings some really exciting capabilities to the table which allows them to solve serious impediments they experience, not just because it is a shiny new thing.
Rhetorical question: Now that we have all this infrastructure for gasoline based cars, is that reason enough to hold back the “churn” to electric cars?
I heard there's some interop between Python and Julia, but I guess it comes with plenty of caveats, like it works for most cases, but will give you enough headache like "oh, of course this use case is not supported like that" etc.
Once you're in Python you can do anything. Spin up a web server, render something in 3D, read obscure data formats...
When I see academic code from 10 years ago written in Matlab I cringe because I know it will be a pain to work with that code. There was a big reason people moved away from Matlab as it cannot support large applications. In Python you can build huge code bases with modular structure, sane classes etc. as opposed to Matlab. And it's free and open source unlike Matlab. So the switch there was worth it. But I don't want all of today's great Python long tail ecosystem to be lost to obsolescence in 5 or 10 years. It's just sad to see so much effort go down the drain.
Unless Python can fix this, by adopting static typing, it's not entirely fit for purpose in the ecosystem niche it has come to dominate. Replacing Python with a better language would be a net productivity boon for developers and scientists almost everywhere.
Unfortunately, this is not too easy to create using Julia. Much easier than Python, sure, but not too easy. Tellingly, the Julia community apparently does not care for static analysis and have not begun to develop the capacity for it. As a consequence, much Julia code is written in a way that is un-analyzable, which digs it deeper into weak typing.
Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare.
I know there's some interoperability between Julia and Python, but a lot of stuff is not interoperable, and there are annoying differences in array indexing etc.
Why can't people just settle on some tech, let it mature, and enjoy reaching higher and higher in capability? Why do we have to always tear things down and start from scratch?
This causes a lot of artificial bit rot, there's a never ending treadmill of just porting well-developed concepts and mature code to slightly changed representations. It's the busywork of tech.
Does the scientific community also want to follow web dev in their insane churn of frameworks and libraries where last week's best practice is laughable ancient stone-age primitiveness today?