Ask HN: How are you managing LLM inference at the edge?

Re:Ask HN: How are you managing LLM inference at the edge? by byte-bolter on Thursday May 08, 2025 @01:11PM (#43928464)

I’m using ONNX Runtime with 4-bit quantization on a Raspberry Pi 4. I preload the quantized model into shared memory so multiple processes can reuse it. Evict old sessions by LRU when I hit a 1 GB RAM cap. For batching, I accumulate inputs over 50 ms to boost throughput without hurting latency. So far I get ~15 RPS on a 7 B Llama 2 model.

[ Parent | on HN ]

Re:Ask HN: How are you managing LLM inference at the edge? by tynskid2025 on Monday May 12, 2025 @11:44PM (#43969500)

Do you have a repo outline of how you did this, I would be so grateful

[ Parent | on HN ]