News for hackers. Stuff that matters.
Re:Show HN: I made an app to use local AI as daily driver by woadwarrior01 on Wednesday February 28, 2024 @07:51AM (#39537195)
I haven't run any specific low level benchmarks, lately. But chunked prefilling and tvm auto-tuned Metal kernels from mlc-llm seemed to make a big differenced, the last time I checked. Also, compared to stock mlc-llm, I use a newer version of metal (3.0) and have a few modifications to make models have a slightly smaller memory and disk footprint, also slightly faster execution. Because unlike the mlc-llm folks, I only care about compatibility with Apple platforms. They support so much more than that in their upstream project.
Re:Show HN: I made an app to use local AI as daily driver by castles on Wednesday February 28, 2024 @08:34AM (#39537668)
thanks, I'll give it a crack