News for hackers. Stuff that matters.

A comment on Petals: Run 100B+ language models at home bit-torrent style. Parent comment

Re:Petals: Run 100B+ language models at home bit-torrent style by Kerbonut on Monday January 02, 2023 @11:05AM (#34219402)
Hi there, thanks for taking the time to answer questions! There are numerous use cases where even a 100x slowdown would be acceptable if it was demonstrable able to process sensitive data. Can you help me understand what kind of a slowdown that is? Could the 10-100x slowdown be overcome by more compute nodes, or would it require the nodes themselves to be 10-100x faster for e.g.?
[ Parent | on HN ]
Re:Petals: Run 100B+ language models at home bit-torrent style by borzunov on Monday January 02, 2023 @01:30PM (#34221656)
If someone wants to process sensitive data and is okay with 10x slowdown, it's better to use offloading. This is another, slower method for running large LMs locally without high-end GPUs, see details here: https://news.ycombinator.com/item?id=34216213

In other words, if Petals nodes became 10-100x slower, Petals would lose its competitive advantage over simpler methods that don't communicate over the Internet.

[ Parent | on HN ]