Any good Node.js parallelization libraries?

Posted by thatxliner on Tuesday January 17, 2023 @12:45PM
from the self-post dept.

Or do I have to use something along the lines of Web Workers or NodeJS Worker Threads? I would prefer it if I had an API similar to Python’s concurrent.futures. What I really want is apply a function to an array in parallel (i.e. some parallel version of .map)

( 5 comments | on HN )

Re:Any good Node.js parallelization libraries? by PaulHoule on Tuesday January 17, 2023 @12:54PM (#34415881)

It is a single threaded runtime so to parallelize a map you will need to serialize the items and ship them to another node process. A bit like dask in Python.

[ Parent | on HN ]

Re:Any good Node.js parallelization libraries? by normalhappy on Tuesday January 17, 2023 @12:53PM (#34415873)

https://www.npmjs.com/package/paralleljs

[ Parent | on HN ]

Re:Any good Node.js parallelization libraries? by thatxliner on Tuesday January 24, 2023 @09:47AM (#34504202)

Tried it, seemed to actually DECREASE my program’s performance. I think it’s because of the overhead of creating thousands of threads…

[ Parent | on HN ]

Re:Any good Node.js parallelization libraries? by rektide on Tuesday January 17, 2023 @01:11PM (#34416203)

Some ground info, Node has worker_threads for this purpose. https://nodejs.org/api/worker_threads.html

I haven't had cause to use it yet, but I very much trust/respect the team behind: https://www.npmjs.com/package/piscina . It's so popular it has well maintained forks, such as tinypool, https://www.npmjs.com/package/tinypool .

Also seeing good npm popularity metrics around: https://www.npmjs.com/package/synckit

[ Parent | on HN ]

Re:Any good Node.js parallelization libraries? by SvenL on Tuesday January 17, 2023 @02:10PM (#34417166)

We used piscina at it was quite good: https://github.com/piscinajs/piscina

While it may not provide parallel map out of the box it will help to implement it a little bit easier than using nodes api.

[ Parent | on HN ]