Petals: Run 100B+ language models at home bit-torrent style

Re:Petals: Run 100B+ language models at home bit-torrent style by thot_experiment on Monday January 02, 2023 @06:12AM (#34216654)

Interesting, how does that work with the multiple GPUs? I'm not familiar with the internal workings of these models, is there anywhere where I can get a brief rundown of how the processing is split. I imagine there can't me much swapping between GPUs as that seems prohibitively slow? How is the model split such that it can be worked on in parallel by multiple GPUs w/o being bottlenecked by IO?

[ Parent | on HN ]

Re:Petals: Run 100B+ language models at home bit-torrent style by borzunov on Monday January 02, 2023 @06:23AM (#34216747)

I think this is a relevant link for you: https://huggingface.co/transformers/v4.9.0/parallelism.html

For large LMs, people usually use tensor-parallelism (TP) or pipeline-parallelism (PP). TP involves lots of communication, but uses all GPUs 100% of the time and works faster. PP requires much less communication, but may keep some GPUs idle while they are waiting for data from others.

Usually, TP is used when you have good communication channels between GPUs (e.g., they are in one data center and connected with NVLink), while PP is used when communication is a bottleneck (like in Petals, where the data is sent over the Internet, which is much slower than NVLink).

[ Parent | on HN ]

Re:Petals: Run 100B+ language models at home bit-torrent style by zone411 on Monday January 02, 2023 @06:25AM (#34216764)

You can read all the gory details here: https://arxiv.org/pdf/2207.00032.pdf

[ Parent | on HN ]

Re:Petals: Run 100B+ language models at home bit-torrent style by nmitchko on Monday January 02, 2023 @10:56AM (#34219261)

You can split the model across devices with huggingface accelerate library.

Check out the infer_auto_memory_map metho which will optimize the model for your configuration (multi gpu, ram, nvme) and then run dispatch model on with that memory map.

[ Parent | on HN ]