Petals: Run 100B+ language models at home bit-torrent style

Re:Petals: Run 100B+ language models at home bit-torrent style by borzunov on Monday January 02, 2023 @05:18AM (#34216312)

A Petals dev here. Indeed, the public swarm should not be used for any kind of sensitive data (we have warnings about that in the instructions). If someone wants to process such data, we recommend to set up a private swarm among the orgs they trust (e.g., a couple of labs/small companies who don't have many GPUs themselves may set up a private swarm and collaborate to process their datasets).

Regarding homomorphic encryption (HE), I'm afraid the current methods to run neural networks in the HE fashion involve 10-100x slowdown, since they are mostly not designed for floating-point operations. We'd love to find a way to do it faster though, since privacy is obviously an important issue for many tasks.

[ Parent | on HN ]

Re:Petals: Run 100B+ language models at home bit-torrent style by Kerbonut on Monday January 02, 2023 @11:05AM (#34219402)

Hi there, thanks for taking the time to answer questions! There are numerous use cases where even a 100x slowdown would be acceptable if it was demonstrable able to process sensitive data. Can you help me understand what kind of a slowdown that is? Could the 10-100x slowdown be overcome by more compute nodes, or would it require the nodes themselves to be 10-100x faster for e.g.?

[ Parent | on HN ]

Re:Petals: Run 100B+ language models at home bit-torrent style by borzunov on Monday January 02, 2023 @01:30PM (#34221656)

If someone wants to process sensitive data and is okay with 10x slowdown, it's better to use offloading. This is another, slower method for running large LMs locally without high-end GPUs, see details here: https://news.ycombinator.com/item?id=34216213

In other words, if Petals nodes became 10-100x slower, Petals would lose its competitive advantage over simpler methods that don't communicate over the Internet.

[ Parent | on HN ]