Unfortunately a reality vague decision of the technique. They say you can train multiple models at once but wouldn't this require N times more memory? They also say that models that require heavy compute won't parallelize well which also seems just a bit absurd because aren't all those shaders all firing off perfectly in parallel to do all the big matrix math?

Anyone have a link to the source paper?

It's paywalled and has no citations...

https://ieeexplore.ieee.org/document/10130297

https://www.computer.org/csdl/journal/tc/5555/01/10130297/1N...

The person who wrote this article doesn't understand what the problem is, and how this paper doesn't address it at all.

The paper has a better way to share one GPU between small training jobs.

The real world problem is training huge models for a long time across many GPUs.

The problem the paper solves is one that is practically irrelevant aside from some niche cases.

The problem to solve is GPU utilization rate. Training a “huge model” on n GPUs will have a cyclical pattern of idle GPUs and GPUs are expensive. This approach is likely scheduling GPU access across t training jobs to insure GPU is always in use. Maximizing GPU utility is a key requirement for training as a service.

The biggest issue is that the failure mode of this approach (one fault hoses the entire pipeline of T jobs) is exactly opposing the goals of its most likely users: Training SaaS providers.