Serving 70B-Scale LLMs Efficiently on Low-Resource Edge Devices [pdf]

Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D... by zackangelo on Thursday October 03, 2024 @11:23AM (#41731626)

I've only read the abstract but they don't mention quantizing the weights or otherwise trying to shrink the model in any way.

They're claiming to be able to efficiently run larger models without loading the entire thing into GPU memory. If they're using the same weights, the same architecture and just using tensor parallel operations to perform the forward pass that would imply no loss in quality.

I'm sure there are trade-offs but they're not clear by just looking at the abstract.

[ Parent | on HN ]

Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D... by tgtweak on Thursday October 03, 2024 @12:38PM (#41732349)

I read it like this too - no drop in weights or model quality just optimizing the lower boundaries of performance when you are splitting from vram to ram to disk (or network).

[ Parent | on HN ]