@tullio C[i,j] := A[i,k] * B[k,j]On the CPU, the situation is much better. With some help from LoopVectorization.jl (which optimises micro-kernels) it will often beat OpenBLAS at matrix multiplication. The best-case scenario is an operation which would otherwise be permutedims plus matrix multiplication, for which it will often be several times faster, by fusing these.
The notation above is shared by some other packages. TensorOperations.jl is always decomposes to known kernels (including on the GPU) and OMEinsum.jl usually does so (with a fallback to loops), both more like einsum. TensorCast.jl is more like einops, just notation for writing reshape/permute/slice operations.
The issue of "slow loops" is entirely self-inflicted by some languages. It's getting ridiculous to still worry about this shit in 2022.
Simple loops can and should be just as fast as vectorized programs. When they are slower, it is 100% due to deliberate decisions by the language dessigners
https://stackoverflow.com/questions/1303182/how-does-blas-ge...
Why turn your back on all the carefully crafted optimizations?
I don't want to turn my back on the beautiful loop notation. For some algorithms, the loop notation is clearer than any "vectorized" version. It is absurd that the language penalizes you for that. Loops are alright.
https://stackoverflow.com/questions/18365073/why-is-numpys-e...
Did I misunderstood your comment?
The optimizer clearly tries to improve the performance, but in many cases, it doesn't seem to change anything. Let's simply multiply some matrices:
x, y = np.random.rand(200, 200, 200), np.random.rand(200, 200, 200)
I can do %timeit x@y
40.3 ms ± 2.52 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
or a naive %timeit np.einsum('bik,bkj->bij',x,y)
1.53 s ± 21.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
But even with optimization, I see %timeit np.einsum('bik,bkj->bij',x,y, optimize=True)
1.54 s ± 10.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
I'm not sure if I'm doing something wrong.`optimize=True` is generally best when there are more than two tensors in the expression.
%timeit (x_jax @ y_jax).block_until_ready()
579 µs ± 4.54 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit jnp.einsum('bik,bkj->bij',x_jax,y_jax, optimize=True).block_until_ready()
658 µs ± 1.38 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit jnp.einsum('bik,bkj->bij',x_jax,y_jax).block_until_ready()
660 µs ± 2.82 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) For example, the optimization of np.einsum has been passed upstream and most of the same features that can be found in this repository can be enabled with numpy.einsum(..., optimize=True).
So numpy should provide the same perf for most cases.
So usually, I rewrite my formulas into messy combinations of broadcasts, transposes and array multiplications. Is there a package or an algorithm that does this conversion automatically? It seems to be a pretty straightforward problem, at least for most expressions I use.