Petals: Run 100B+ language models at home bit-torrent style

Re:Petals: Run 100B+ language models at home bit-torrent style by bogwog on Monday January 02, 2023 @10:14AM (#34218735)

So NVLink/NVSwitch pools multiple GPU resources on a single (very expensive) system. A cheaper alternative to that is "offloading", which is a technique that splits the inference process into smaller steps, so it can run on systems with much less resources available... and Petals is a 10x faster alternative to that.

Did I get that right?

This AI stuff is moving very fast and it's hard to keep up, but it's all fascinating.

[ Parent | on HN ]

Re:Petals: Run 100B+ language models at home bit-torrent style by parentheses on Monday January 02, 2023 @10:44AM (#34219104)

Matches my understanding also. Can someone in the know confirm?

[ Parent | on HN ]

Re:Petals: Run 100B+ language models at home bit-torrent style by nmitchko on Monday January 02, 2023 @10:53AM (#34219218)

Your understanding is correct, but I can't vouch for the claim's accuracy. This could make the execution of models much more accessible to people who don't have a 4x / RTX3090 or better in a ML or mining rig ..

[ Parent | on HN ]

Re:Petals: Run 100B+ language models at home bit-torrent style by cardine on Monday January 02, 2023 @01:02PM (#34221230)

Offloading is when the computation is done on the CPU instead of the GPU. DeepSpeed is an example of this.

[ Parent | on HN ]

Re:Petals: Run 100B+ language models at home bit-torrent style by borzunov on Monday January 02, 2023 @02:02PM (#34222160)

In case of offloading, the computations are usually still performed on GPU, but the model is hosted in RAM/SSD instead of the GPU memory (and its chunks are copied to the GPU memory when necessary).

[ Parent | on HN ]

Re:Petals: Run 100B+ language models at home bit-torrent style by cardine on Monday January 02, 2023 @03:01PM (#34222878)

A lot of computation is offloaded to the CPU, such as gradients and optimizer states. You are right though that quite a bit of computation is still done on the GPU.

[ Parent | on HN ]

Re:Petals: Run 100B+ language models at home bit-torrent style by rolenthedeep on Monday January 02, 2023 @04:53PM (#34224181)

I remember when GPUs were starting to support arbitrary computation and offloading meant shifting work away from the CPU.

[ Parent | on HN ]

Re:Petals: Run 100B+ language models at home bit-torrent style by borzunov on Monday January 02, 2023 @01:20PM (#34221490)

You're right. This comment explains what offloading is in more detail: https://news.ycombinator.com/item?id=34216213

[ Parent | on HN ]