I do as much data work as possible in data.table, which parallelizes things under the hood, and doesn't seem to require me to configure anything.
Re:Test for lists in Cython
by curiousgal on Monday April 19, 2021 @07:41AM
(#26861607)
I haven't run into any issues with that package honestly (mclapply ftw!). Other than not being able to use it on Windows of course.
It’s process based, so communication overhead is heavy and the infrastructure is minimal - if you’re distributing jobs which are individually large it works fine, but it can’t really be used to replace multithreading in a shared memory model.
Same thing in the R ecosystem, it's possible to use multiple cores with the parallel package but there are caveats too.