News for hackers. Stuff that matters.
Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by 19h on Saturday March 11, 2023 @10:09AM (#35108835)
I couldn't get the mps python version to run, it's insane how much setup it requires... I don't think the Gerganov C++ version uses CoreML or Neural Engine.

I previously tried to play with ANE based off prior reverse engineering work [0] but couldn't get it to work nicely. It's actually beyond me how Gerganov's version performs so well -- the output quality of the non-quantised version running on A100 (AWS) isn't noticably better than the one I'm getting.

[0] https://i.blackhat.com/asia-21/Friday-Handouts/as21-Wu-Apple...

There are no comments on this comment yet.