News for hackers. Stuff that matters.
Re:Piscina – The Node.js Worker Pool by freeqaz on Tuesday March 23, 2021 @03:31PM (#26559317)
Does anybody use this or anything similar? If so, what problems are you solving?
Re:Piscina – The Node.js Worker Pool by searchableguy on Tuesday March 23, 2021 @05:12PM (#26560464)
I use workers in deno to evaluate user facing code which may take a lot of time to finish and need sandboxing. Another place it's used in where I want to keep context and reuse it for subsequent code execution for each user.

In deno, you can restrict what the code inside worker can do by passing a map of allowed permissions. I have built a simple privilege system on top of this to allow users different access level.

This is cheaper and faster than spinning up container.

Re:Piscina – The Node.js Worker Pool by domenicd on Tuesday March 23, 2021 @05:13PM (#26560473)
I maintain a command-line utility which assembles an eBook from scraped .xhtml files. It uses a similar package, workerpool, to process multiple chapters in parallel.

https://github.com/domenic/worm-scraper/blob/master/lib/conv...

https://www.npmjs.com/package/workerpool

Re:Piscina – The Node.js Worker Pool by Etheryte on Tuesday March 23, 2021 @03:48PM (#26559534)
The worker API in Javascript is quite a pain to use, but needless to say multithreading is invaluable in many contexts, both in the browser and in Node. I haven't used this library but it seems to solve a similar problem to other ones in the same space — make writing multithreaded code sane, allowing you to avoid writing a bunch of repetitive boilerplate.
Re:Piscina – The Node.js Worker Pool by rektide on Tuesday March 23, 2021 @04:06PM (#26559749)
Knex, the SQL query builder, uses Tarn.js[1] for connection pooling to the db.

I've been using Tarn a bunch at work recently. We're doing some batch jobs, and I'm queuing work at each stage in Tarn.js pools. I created my own enqueue function that waits until the pool is less than a high-water mark in size before enqueueing. Then the pool has however many workers running.

Neither of these are off-thread pools. But they help a lot of for managing multiple async operations.

[1] https://github.com/vincit/tarn.js/

Re:Piscina – The Node.js Worker Pool by curben on Tuesday March 23, 2021 @03:44PM (#26559475)
AFAIK this is a wrapper of worker_threads API. https://github.com/tuananh/camaro utilises this for multi-threading processing of XML input. Hexo (static site generator) is attempting to utilise this library (https://github.com/hexojs/hexo/issues/4355).
Re:Piscina – The Node.js Worker Pool by vorticalbox on Tuesday March 23, 2021 @05:52PM (#26560845)
At work I use https://github.com/pioardi/poolifier

We have an api used for generating reports from mongo dB to csv, this let's the report process in the background leaving the report api to still handle requests.

Re:Piscina – The Node.js Worker Pool by niftylettuce on Tuesday March 23, 2021 @04:01PM (#26559688)
Yeah. I built Bree @ https://jobscheduler.net

It has support for worker threads + cron + human strings

GH: https://github.com/breejs/bree

Re:Piscina – The Node.js Worker Pool by ddoolin on Tuesday March 23, 2021 @04:09PM (#26559792)
This is cool. Our use case was very simple so we went with node-schedule but this may have more upsides. I've had trouble with the former. Thanks.
Re:Piscina – The Node.js Worker Pool by niftylettuce on Tuesday March 23, 2021 @05:23PM (#26560581)
I helped maintain all the Node.js node/cron/agenda libraries. Just switch to Bree and your troubles will be gone.
Re:Piscina – The Node.js Worker Pool by ddoolin on Tuesday March 23, 2021 @04:00PM (#26559674)
We use worker threads directly to process large unorganized (for the browser) datasets and do some deductions before it hits the store.

We also have a worker thread blocked on a redis channel that acts as a queue.

Re:Piscina – The Node.js Worker Pool by Skhalar on Tuesday March 23, 2021 @03:51PM (#26559571)
This is useful for processing large chunks of data like audio files (look at Superpowered sdk) but breaking em down or when processing multiple files.
Re:Piscina – The Node.js Worker Pool by barefeg on Tuesday March 23, 2021 @03:37PM (#26559388)
I’m guessing the fact that node is single threaded
Re:Piscina – The Node.js Worker Pool by nobleach on Tuesday March 23, 2021 @04:28PM (#26559994)
JavaScript is single threaded, Node is most certainly not.
Re:Piscina – The Node.js Worker Pool by gbrits on Tuesday March 23, 2021 @06:20PM (#26561073)
Yes it surely is
Re:Piscina – The Node.js Worker Pool by zbentley on Tuesday March 23, 2021 @06:50PM (#26561349)
My "htop" display surely disagrees. So do my node programs that lag out waiting for disk IO thread pool slots. Sounds does node's documentation: https://nodejs.org/api/worker_threads.html
Re:Piscina – The Node.js Worker Pool by hmcdona1 on Tuesday March 23, 2021 @06:50PM (#26561352)
It's not exactly. Certain operations like I/O are threaded in Node. libuv has it's own threadpool that it uses for a lot of these types of tasks.
Re:Piscina – The Node.js Worker Pool by jayflux on Wednesday March 24, 2021 @04:52AM (#26564903)
Node uses a thread pool for tasks like reading files async and crypto operations etc, you can read about it here https://nodejs.org/en/docs/guides/dont-block-the-event-loop/
Re:Piscina – The Node.js Worker Pool by maxrev17 on Tuesday March 23, 2021 @03:45PM (#26559482)
Might be quite nice for keeping tokens refreshed?