Re:Petals: Run 100B+ language models at home bit-torrent style
by colordrops on Monday January 02, 2023 @04:46AM
(#34216143)
Why not?
[ Parent
| on HN ]
Re:Petals: Run 100B+ language models at home bit-torrent style
by simongray on Monday January 02, 2023 @05:05AM
(#34216233)
How would one make a reliable realtime system that depends entirely on unknown network conditions? Perhaps inside a closed network it is possible.
Re:Petals: Run 100B+ language models at home bit-torrent style
by _joel on Monday January 02, 2023 @05:07AM
(#34216247)
That's orthoganal to a realtime system. You can infer at a fair speed so realtime would be possible.
Re:Petals: Run 100B+ language models at home bit-torrent style
by simongray on Monday January 02, 2023 @05:26AM
(#34216369)
Guarantees are not orthogonal to realtime feedback, they are essential. If I write a query, it is not irrelevant whether it takes 1 second or 1 minute to return at any given moment.
You write that speed can be inferred, but the analogy that was used here is BitTorrent—and my experience with BitTorrent tells me that it certainly cannot be inferred.
Re:Petals: Run 100B+ language models at home bit-torrent style
by _joel on Monday January 02, 2023 @05:29AM
(#34216392)
If you read the article text and the response from the dev then yes, inference can happen at 1/s or if parallelised, more. I'm not sure what your parameters are for a realtime system. If you're talking about network reliability, that's a different issue. Yes it can infer quickly, can it do it reliably is another matter.