My favourite approach assigns one global thread pool (queue) to each function or processing step. The arguments to the functions will change, but the code is always the same, so the resource requirements are also well understood.

I think I get what he's saying. To avoid one stage in the multi-step computation acquiring too many resources and starving other stages:

- Cap the resources used by each stage.

- Create one threadpool per stage.

- Include sufficient information in pending jobs that the threadpool can calculate the resources needed to process each job.

- Only start a job running when it can be run without exceeding resource cap for the stage.

Conceptually, you can think of it as pre-allocating a bundle of resources to each processing stage: threads, memory, etc. Each processing stage then runs as many jobs concurrently as it can with the allocated bundle of resources. That's the idea, but in reality each job allocates its own resources from a common pool that is shared among all the processing stages, so the processing stage has to calculate how much memory, etc. each job will allocate before giving it a thread to run on.

For a software pipeline, you don't need to hand data off to a different thread, you just need to make sure the data is run with the right execution. That could be on any thread, and the execution/function can be run on any number of threads at the same time as long as the data is separate.