News for hackers. Stuff that matters.

A comment on The Coming of Local LLMs.

Re:The Coming of Local LLMs by anentropic on Tuesday April 11, 2023 @11:58AM (#35527014)
Apple should get working on a version of the Neural Engine that is useful for these models, and remove the 3GB size limit [1] to take full advantage of the 'unified' memory architecture. Game changer.

Waste of die space currently (on Macbook at least, I'm sure they find uses for it in the iPhone)

[1] https://github.com/smpanaro/more-ane-transformers/blob/main/...

[ Parent | on HN ]
Re:The Coming of Local LLMs by leetharris on Wednesday April 12, 2023 @09:05AM (#35539005)
It's not a waste on Mac, it will dynamically switch between GPU and NPU whenever CoreML is called. There are a decent amount of applications that use CoreML.

But I do agree it should be improved!

[ Parent | on HN ]