Re:Petals: Run 100B+ language models at home bit-torrent style
by nmitchko on Monday January 02, 2023 @10:56AM
(#34219261)
You can split the model across devices with huggingface accelerate library.
Check out the infer_auto_memory_map metho which will optimize the model for your configuration (multi gpu, ram, nvme) and then run dispatch model on with that memory map.
Check out the infer_auto_memory_map metho which will optimize the model for your configuration (multi gpu, ram, nvme) and then run dispatch model on with that memory map.