News for hackers. Stuff that matters.
Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by cypress66 on Thursday October 12, 2023 @01:14AM (#37853666)
>I'm running gpt2-xl (1.5B params) locally with KV caching at 120ms/token (vs. 450ms without caching).

That seems very slow compared to llama cpp?

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by smpanaro on Thursday October 12, 2023 @04:22AM (#37854736)
Yeah, I believe it is. You trade off speed for lower power usage and CPU. 8 tokens/sec is usable though.