Re:The Coming of Local LLMs
by beiller on Tuesday April 11, 2023 @12:56PM
(#35527856)
Which LLM can run on apple's neural cores / GPU cores? I can only run on plain ol' CPU cores (llama), and it runs fine on my Ryzen CPU for less than half the price of that system. That being said I'm switching from Ubuntu to Arch cause I'm sick of all my packages being way out of date!
[ Parent
| on HN ]
Re:The Coming of Local LLMs
by ingenieroariel on Tuesday April 11, 2023 @01:32PM
(#35528338)
tinygrad
https://github.com/geohot/tinygrad/tree/master/accel/ane
But I have not tested it on Linux since Asahi has not yet added support.
Same machine but OSX, llama.cpp runs at 18ms per token (7B) and 200ms per token (65B) on CPU using float16.