Ask HN: How are you managing LLM inference at the edge?

Posted by gray_amps on Thursday May 08, 2025 @01:06PM
from the ask-hn dept.

I’m building a system to run small LLMs on-device (mobile, IoT, on-prem servers) and would love to hear how others have tackled the challenges.

Context:

Use cases: offline chatbots, smart cameras, local data privacy

Models: 7–13B parameter quantized models (e.g. Llama 2, Vicuna)

Constraints: limited RAM/flash, CPU-only or tiny GPU, intermittent connectivity

Questions:

What runtimes or frameworks are you using (ONNX Runtime, TVM, custom C++)?

How do you handle model loading, eviction, and batching under tight memory?

Any clever tricks for quantization, pruning, or kernel fusions that boost perf?

How do you monitor and update models securely in the field?

Looking forward to your benchmarks, war stories, and code pointers!

( 2 comments | 4 points | on HN )

Re:Ask HN: How are you managing LLM inference at the edge? by byte-bolter on Thursday May 08, 2025 @01:11PM (#43928464)

I’m using ONNX Runtime with 4-bit quantization on a Raspberry Pi 4. I preload the quantized model into shared memory so multiple processes can reuse it. Evict old sessions by LRU when I hit a 1 GB RAM cap. For batching, I accumulate inputs over 50 ms to boost throughput without hurting latency. So far I get ~15 RPS on a 7 B Llama 2 model.

[ Parent | on HN ]

Re:Ask HN: How are you managing LLM inference at the edge? by tynskid2025 on Monday May 12, 2025 @11:44PM (#43969500)

Do you have a repo outline of how you did this, I would be so grateful

[ Parent | on HN ]