Re:Python’s Weak Performance Matters
by pletnes on Wednesday February 07, 2018 @02:54AM
(#16322897)
That’s easy with python, too, in a lot of number crunching cases. Numpy with MKL will use all your cores, as will e.g dask and other libraries built on numpy. Farming out embarassingly parallel work to threads or processes is also easy.
[ Parent
| on HN ]
Re:Python’s Weak Performance Matters
by luispedrocoelho on Wednesday February 07, 2018 @03:25AM
(#16323008)
If I can fit the code into numpy-like structure, then Python is typically fine.
The issue is when I cannot.
Re:Python’s Weak Performance Matters
by fnord123 on Wednesday February 07, 2018 @08:33AM
(#16324232)
Then move the function to a pyx file and build it with Cython. Problem solved.
Re:Python’s Weak Performance Matters
by kalefranz on Wednesday February 07, 2018 @05:00AM
(#16323382)
Also look into numba as a jit decorator for python functions.
Have you given dask a try? It gives you out-of-core arrays with numpy semantics and distributed computing.
Re:Python’s Weak Performance Matters
by VHRanger on Wednesday February 07, 2018 @08:50AM
(#16324337)
Dask doesn't solve that problem since it's a wrapper around pandas functions.
If you can't make the core pandas code decently fast, dask won't save you.
Re:Python’s Weak Performance Matters
by kiriliponi on Wednesday February 07, 2018 @09:43AM
(#16324675)
dask.dataframe might not help but dask.distributed could in that case.
I've had success using it on non vanilla stuff (i.e. code that could not get converted to play natively with numpy/pandas structures)
As a bonus, the nice profiling tools (built within dask) have also helped me improve the performance of the code.
There’s dask.array which works on numpy arrays instead of dataframes. Otherwise, your argument holds.