News for hackers. Stuff that matters.

A comment on Petals: Run 100B+ language models at home bit-torrent style. Parent comment

Re:Petals: Run 100B+ language models at home bit-torrent style by simongray on Monday January 02, 2023 @05:26AM (#34216369)
Guarantees are not orthogonal to realtime feedback, they are essential. If I write a query, it is not irrelevant whether it takes 1 second or 1 minute to return at any given moment.

You write that speed can be inferred, but the analogy that was used here is BitTorrent—and my experience with BitTorrent tells me that it certainly cannot be inferred.

[ Parent | on HN ]
Re:Petals: Run 100B+ language models at home bit-torrent style by _joel on Monday January 02, 2023 @05:29AM (#34216392)
If you read the article text and the response from the dev then yes, inference can happen at 1/s or if parallelised, more. I'm not sure what your parameters are for a realtime system. If you're talking about network reliability, that's a different issue. Yes it can infer quickly, can it do it reliably is another matter.
[ Parent | on HN ]