Re:Petals: Run 100B+ language models at home bit-torrent style
by _joel on Monday January 02, 2023 @05:07AM
(#34216247)
That's orthoganal to a realtime system. You can infer at a fair speed so realtime would be possible.
[ Parent
| on HN ]
Re:Petals: Run 100B+ language models at home bit-torrent style
by simongray on Monday January 02, 2023 @05:26AM
(#34216369)
Guarantees are not orthogonal to realtime feedback, they are essential. If I write a query, it is not irrelevant whether it takes 1 second or 1 minute to return at any given moment.
You write that speed can be inferred, but the analogy that was used here is BitTorrent—and my experience with BitTorrent tells me that it certainly cannot be inferred.
Re:Petals: Run 100B+ language models at home bit-torrent style
by _joel on Monday January 02, 2023 @05:29AM
(#34216392)
If you read the article text and the response from the dev then yes, inference can happen at 1/s or if parallelised, more. I'm not sure what your parameters are for a realtime system. If you're talking about network reliability, that's a different issue. Yes it can infer quickly, can it do it reliably is another matter.