News for hackers. Stuff that matters.
Re:A brief history of LLaMA models by simonw on Saturday April 29, 2023 @05:41PM (#35757499)
My understanding is that part of it is that Apple Silicon shares all available RAM between CPU and GPU.

I'm not sure how many of these models are actively taking advantage of that architecture yet though.

Re:A brief history of LLaMA models by int_19h on Saturday April 29, 2023 @06:55PM (#35758091)
The GPU isn't actually used by llama.cpp. What makes it that much faster is that the workload, either on CPU or on GPU, is very memory-intensive, so it benefits greatly from fast RAM. And Apple is using DDR5 running at very high clock speeds for this shared memory stuff.

It's still noticeably slower than GPU, though.