News for hackers. Stuff that matters.

A comment on Show HN: sllm – Split a GPU node with other developers, unlimited tokens. Parent comment

Re:Show HN: sllm – Split a GPU node with other developers, u... by kaoD on Saturday April 04, 2026 @03:15PM (#47642303)
> The model weights stay resident in VRAM permanently so there's no loading/unloading per request.

Yes, I was thinking about context buffers, which I assume are not small in large models. That has to be loaded into VRAM, right?

If I keep sending large context buffers, will that hog the batches?

[ Parent | on HN ]
Re:Show HN: sllm – Split a GPU node with other developers, u... by jrandolf on Saturday April 04, 2026 @04:14PM (#47642905)
Not if you are the only one. We have rate limits to prevent this in case, idk, you share your key with 1000 people lol.
[ Parent | on HN ]