News for hackers. Stuff that matters.

A comment on Bringing K/V context quantisation to Ollama. Parent comment

Re:Bringing K/V context quantisation to Ollama by seb314 on Thursday December 05, 2024 @09:43AM (#42328666)
For running llms _locally_ on Android, there's "pocketpal" (~7tok/s on a pixel 7 pro for some quant of llama 3.2 3B).

(Not sure if it uses ollama though)

[ Parent | on HN ]

There are no comments on this comment yet.