News for hackers. Stuff that matters.

A comment on Petals: Run 100B+ language models at home bit-torrent style. Parent comment

Re:Petals: Run 100B+ language models at home bit-torrent style by borzunov on Monday January 02, 2023 @02:02PM (#34222160)
In case of offloading, the computations are usually still performed on GPU, but the model is hosted in RAM/SSD instead of the GPU memory (and its chunks are copied to the GPU memory when necessary).
[ Parent | on HN ]
Re:Petals: Run 100B+ language models at home bit-torrent style by cardine on Monday January 02, 2023 @03:01PM (#34222878)
A lot of computation is offloaded to the CPU, such as gradients and optimizer states. You are right though that quite a bit of computation is still done on the GPU.
[ Parent | on HN ]