Petals: Run 100B+ language models at home bit-torrent style

Re:Petals: Run 100B+ language models at home bit-torrent style by borzunov on Monday January 02, 2023 @05:07AM (#34216250)

A Petals dev here. It is not real-time, but we think the speed of ~1 token/sec may be enough for some interactive apps such as chat bots (especially, if you show tokens to a user once they are generated). You can try one at http://chat.petals.ml (heads-up: it may be laggy right now due to lots of HN users trying out the system).

Of course, you could do better if you have enough high-end GPUs to host the entire model yourself (3x A100 or 8x 3090). But if you don't, 1 token/sec is much faster than what you get with other existing methods.

[ Parent | on HN ]

Re:Petals: Run 100B+ language models at home bit-torrent style by dpflan on Monday January 02, 2023 @08:37AM (#34217672)

I have not read the technical details, apologies for ignorance, but is there an opportunity for caching?

[ Parent | on HN ]

Re:Petals: Run 100B+ language models at home bit-torrent style by jerpint on Monday January 02, 2023 @12:25PM (#34220651)

Probably not, since you need to compute the activations of unknown inputs and there could be infinitely many variations of them

[ Parent | on HN ]

Re:Petals: Run 100B+ language models at home bit-torrent style by KaoruAoiShiho on Monday January 02, 2023 @10:14AM (#34218726)

What are the speeds of other existing methods?

[ Parent | on HN ]

Re:Petals: Run 100B+ language models at home bit-torrent style by borzunov on Monday January 02, 2023 @02:15PM (#34222309)

Theoretical best-case for RAM offloading is 5.5 sec/token, for SSD offloading - 22 sec/token. Implementations we've tested are not faster than 10 sec/token though. See details in our paper: https://arxiv.org/pdf/2209.01188.pdf

[ Parent | on HN ]