News for hackers. Stuff that matters.

A comment on Petals: Run 100B+ language models at home bit-torrent style. Parent comment

Re:Petals: Run 100B+ language models at home bit-torrent style by KaoruAoiShiho on Monday January 02, 2023 @10:14AM (#34218726)
What are the speeds of other existing methods?
[ Parent | on HN ]
Re:Petals: Run 100B+ language models at home bit-torrent style by borzunov on Monday January 02, 2023 @02:15PM (#34222309)
Theoretical best-case for RAM offloading is 5.5 sec/token, for SSD offloading - 22 sec/token. Implementations we've tested are not faster than 10 sec/token though. See details in our paper: https://arxiv.org/pdf/2209.01188.pdf
[ Parent | on HN ]