Re:Show HN: sllm – Split a GPU node with other developers, u...
by ninjha on Saturday April 04, 2026 @01:22PM
(#47641157)
> how many work units can run in parallel
not original author but batching is one very important trick to make inference efficient, you can reasonably do tens to low hundreds in parallel (depending on model size and gpu size) with very little performance overhead
not original author but batching is one very important trick to make inference efficient, you can reasonably do tens to low hundreds in parallel (depending on model size and gpu size) with very little performance overhead