Re:The Unscalable, Deadlock-Prone, Thread Pool
by chrisseaton on Tuesday February 26, 2019 @09:42PM
(#19260422)
Right - we need holistic scheduling systems that balance all resources and tasks. For example scheduling an IO heavy-task alongside a compute-heavy task and letting them both run well, rather than picking two compute-heavy tasks at the same time, and only scheduling as many tasks as there is RAM, IO, and other resource capacity for to use productively. Lots of people are trying because at data-centre scale and with serverless that lets you run code wherever they want this could be a huge advantage... not sure how much success yet.
[ Parent
| on HN ]
Re:The Unscalable, Deadlock-Prone, Thread Pool
by beached_whale on Tuesday February 26, 2019 @10:07PM
(#19260520)
I wonder if something as simple as tagging would give a huge benefit towards that. I should know if my job uses lots of memory(often I know exactly how much), or IO, or CPU.
Re:The Unscalable, Deadlock-Prone, Thread Pool
by ahartmetz on Wednesday February 27, 2019 @03:09AM
(#19261749)
This is something I have pondered for the Bitbake build system. It compiles a Linux "distribution" for embedded systems and uses a lot of CPU, RAM, and I/O. Often one is starving the other. I use a spinning disk for build dirs currently because they use a lot of space. Even with an SSD there is room for better scheduling.