Show HN: We made our own inference engine for Apple Silicon

Re:Show HN: We made our own inference engine for Apple Silicon by giancarlostoro on Tuesday July 15, 2025 @02:26PM (#44574335)

Hoping the author can answer, I'm still learning about how this all works. My understanding is that inference is "using the model" so to speak. How is this faster than established inference engines specifically on Mac? Are models generic enough that if you build e.g. an inference engine focused on AMD GPUs or even Intel GPUs, would they achieve reasonable performance? I always assumed because Nvidia is king of AI that you had to suck it up, or is it just that most inference engines being used are married to Nvidia?

I would love to understand how universal these models can become.

[ Parent | on HN ]

Re:Show HN: We made our own inference engine for Apple Silicon by darkolorin on Tuesday July 15, 2025 @04:54PM (#44575711)

Basically “faster” means better performance e.g. tokens/s without loosing quality (benchmarks scores for models). So when we say faster we provide more tokens per second than llama cpp. That means we effectively utilize hardware API available (for example we wrote our own kernels) to perform better.

[ Parent | on HN ]