Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D...
by _ache_ on Thursday October 03, 2024 @12:20PM
(#41732139)
It's not disk swap. It's multi-devices LLM.
[ Parent
| on HN ]
Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D...
by wpietri on Thursday October 03, 2024 @01:09PM
(#41732656)
I think the point is that it has the same sort of latency tradeoff that disk swap did: it's awful, but sometimes better than nothing.
Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D...
by kridsdale3 on Thursday October 03, 2024 @01:09PM
(#41732651)
That looked like an analogy. Back in the days of a mechanical arm moving magnetic fields around in our PCs, you could have the illusion of infinite RAM as long as you're ok with microsecond operations now taking two million times longer. This is akin.