Serving 70B-Scale LLMs Efficiently on Low-Resource Edge Devices [pdf]

Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D... by thelastparadise on Thursday October 03, 2024 @11:52AM (#41731855)

Is there any predictability/patterns for neuron/layer activation? If so, would it be reasonable to have a second tiny model that specifically tries to predict activation and preemptively swap those into memory?

[ Parent | on HN ]

Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D... by tcdent on Thursday October 03, 2024 @12:01PM (#41731946)

Depends on the architecture, but generally you just move through the layers linearly. Simple iteration.

The number of layers, and the amount of time spent in each of them, makes me think any benefit from pre-loading the layer ahead is negligible.

You really need the entire model on device to consider it performant.

[ Parent | on HN ]

Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D... by miki123211 on Thursday October 03, 2024 @12:04PM (#41731966)

This isn't how neural networks work.

For vanilla models, you always use all the weights. That isn't true for mixture-of-experts, though, and in that setting, your approach has merit.

[ Parent | on HN ]