Re:Run 100B+ language models at home, BitTorrent‑style
by winddude on Monday March 20, 2023 @09:57PM
(#35241614)
I got worse than 1 token/sec, and yes, wasn't impressed with bloom results, but I believe it's also very foreign language heavy. I haven't tried it yet but I believe flexGen benchmarked faster as well.
[ Parent
| on HN ]
Re:Run 100B+ language models at home, BitTorrent‑style
by borzunov on Tuesday March 21, 2023 @07:24AM
(#35244794)
A Petals dev here. FlexGen is good at high-throughput inference (generating multiple sequences in parallel). During single-batch inference, it spends more than 5 sec/token in case of GPT-3/BLOOM-sized models.
So, I believe 1 sec/token with Petals is the best you can get for the models of this size, unless you have enough GPUs to fit the entire model into the GPU memory (you'd need 3x A100 or 8x 3090 for the 8-bit quantized model).