Performance of llama.cpp on Apple Silicon A-series

Re:Performance of llama.cpp on Apple Silicon A-series by andy99 on Tuesday December 19, 2023 @07:33PM (#38704005)

I agree of course though I also think there is/was a lot of "low hanging fruit" in inference. Pytorch is really a model training framework and pytorch/python have not traditionally been well suited to fast inference, e.g. with lacking quantization but also lacking optimized inference code. That's changing, for example this recent post: https://pytorch.org/blog/accelerating-generative-ai-2/

But "AI" is still very research focused (training) and not application focused (inference). Llama.cpp et al are filling in this gap.

[ Parent | on HN ]

Re:Performance of llama.cpp on Apple Silicon A-series by carterschonwald on Wednesday December 20, 2023 @07:14PM (#38715469)

I honestly would love to full time focus on making high performance tools for ml / applied math, but its a hard area to break into, or at least I’ve been going about it wrong.

[ Parent | on HN ]