News for hackers. Stuff that matters.
Re:Run 70B LLM Inference on a Single 4GB GPU with This New T... by rthnbgrredf on Sunday December 03, 2023 @02:38PM (#38510020)
This is interesting. While I think this approach may not be practical for a single computer due to poor performance, it could open new possibilities for distributed computing. Here, users across the internet could pool their graphics cards and the 80 layers could be distributed across 80 GPUs, with each GPU handling one layer. The results would then be processed through the system. Admittedly, if there is an average latency of 50 ms for each of the 80 GPUs, it would result in an additional 4 seconds of network delay. However, the overall process could be up to 80 times faster than on a single GPU since the layers are already in the GPU memory, albeit on different cards. This method also has the potential to enable the execution of much larger models on consumer hardware.
Re:Run 70B LLM Inference on a Single 4GB GPU with This New T... by onlywicked on Sunday December 03, 2023 @02:48PM (#38510093)
There is already an implementation along the same line using the torrent architecture.

https://petals.dev/

Re:Run 70B LLM Inference on a Single 4GB GPU with This New T... by alchemist1e9 on Sunday December 03, 2023 @05:35PM (#38511453)
Does any one know the amount of data that is passed between layers? specifically the data size as this information determines the viability of distributed inference.