The implementation may still be a bit buggy though, so use with caution! https://bugs.python.org/issue38119
The main selling point of the library is automatic differential and compilation to XLA, but I've been using it even when I don't need gradients, as it's really fast (due to compilation). I also really like the random number generator as it's very good for reproducibility.
I've played around with Julia in the past and really liked it, but in terms speed Jax has pretty much solved that problem for me
You may find this interesting https://github.com/scikit-hep/iminuit/blob/develop/tutorial/...
It takes a bit of getting used to but the performance gains on impressive. Basically, my bottlenecks shift from compute to i/o.
I think you have to balance it against writing in C/C++. Mentally, it is basically the same work as writing in C (you manage memory/you write complicated for-loops) but you have good array support with numpy. The primary advantage for me that everything stays in the python runtime environment. You just run the code without any extra steps.
...
What is missing from the timing type 'toy' benchmarks is an understanding that there is typically more than one bottleneck in a real problem and it is easy to choose the wrong one to optimize and get little gains.
After starting C (30 years ago now), spending a long time in C#, then switching to Python a few years ago, I think the unappreciated advantage of python is that I have to abandon all pretense of caring about speed and just get stuff working. It basically solves the pre-mature optimization problem for me by being a fast interpreted language rather than a slow compiled language.
I feel the same way. With Python I just write the simplest algorithm that first comes to my mind, even though I know that it is not the most optimized way of doing things. But most of the time I am surprised that it works so fast that I realize I actually don't need to optimize it.
And being able to create and easily manipulate dictionaries and tuples also allows me to create efficient data structures very quickly.
Edit: also note that a big part of the umap library is written in numba.
TBF I am mainly using it for mostly pure path functions.
With Jax you can write native for loops that can also be jitted (I imagine you can also do this in numba?); this can then be really fast. Though in that case you would have to write the optimisation algorithm yourself which is not always practical!
Another big speedup in Jax is due to vmap/pmap, which allow to vectorise/parallelise computation. For example you can build a massive gram matrix really quickly using vmap.
Another point: Jax can also run on GPU (like numba :) ) without having to rewrite anything.
Yes, you can just pass the function and it runs the optimization, in severe cases, you can start by doing a grid `.scan` or `.simplex` (Nelder-Mead simplex method), then `migrad` to minimize, and `.hesse` for 1 sigma bound.
You can also provide a gradient function that, well computes the gradient instead of computing it numerically.
> I imagine you can also do this in numba
Yes! It compiles to native code.
> Another big speedup in Jax is due to vmap/pmap, which allow to vectorise/parallelise computation.
It's possible to compile some vmapped functions with numba too, if the link is any indication, you may see even greater speedup than just jax.grad
However, I do concur that jax's vmap is absolutely fantastic and I found it very useful on many occasions.
Dask is great if you’re processing large amounts of data, and it recommends and supports threads for this reason.
Not sure there’s anything to prevent you from foot-shooting if you’re operating on the same array. I’d expect that this is only safe to do if, as you say, arrays are not overlapping. Let me know what you find.
I understand the sentiment, but in this case I wonder if it is less a python issue and more an issue with numpy. I don't see any reason why numpy couldn't execute the last line of the following multi-threaded:
>>> import numpy as np
>>> a = np.arange(1000000000)
>>> b = 2 \* a
The same thing applies to matrix multiplication. I'm guessing numpy has chosen not to for good reason though.edit: Actually it seems numpy is multi-threaded under certain circumstances:
https://stackoverflow.com/questions/16617973/why-isnt-numpy-...
Distribute your data and run everything as dask.delayed and then compute only at the end.
Also check out legate.numpy from Nvidia which promises to be a drop in numpy replacement that will use all your CPU cores without any tweaks on your part.
Also a big downside with JavaScript. Of course both Python and JS are high-level interpreted languages where high-performance use cases aren't the foremost priority.
Can't Node run multiple processes (or multiple workers, not sure of the proper terminology) on multiple cores? As discussed in this thread on StackOverflow: https://stackoverflow.com/questions/61893497/node-js-on-mult...
Of course, but you can also run multiple Python processes.
Modern JS engines are all primarily JIT-based, generally only interpreting code when it's faster than waiting on the JIT. JS JITs are quite good now, and in many cases will produce optimized C-equivalent compiled code from fairly naive JS.
This can be an absolute sinkhole for performance, but if you have a large dataset you shouldn't be sharing lines that much one would hope.
Same thing in the R ecosystem, it's possible to use multiple cores with the parallel package but there are caveats too.
Also, if you are running on Intel you might have some luck with Intel's profiling tools as, although I've never used them for python they are by far and away the best things in this area, they do support Python.
I don't know what details they give you in from Python code directly, but if you've used perf to profile code at a low level before, vTune is like a Tom & Jerry shotgun in comparison.
Never seen this before. Thanks!
There is multiprocessing module but you cannot share memory between processes. I guess the best options are either writing a C extension or using Numba. Writing C extensions require either distributing binary packages or C compiler to be present on the target computer which is not always ideal. So is using Numba the best solution currently? I tried it a bit in the past but the errors I got was a bit hard to interpret compared to regular Python errors.
I wish the multithreading module had support for native threads. Is there any PEPs trying to bring easy multi-core support for Python?