Likely 1/4th the size, 1/4th the power consumption, and 4x the ram bandwidth. Have you by chance played with any LLMs? Just saw a post that someone managed 5 tokens/sec with the llama 65B model.
Re:Apple unveils new Mac Studio and brings Apple Silicon to ...
by kamranjon on Monday June 05, 2023 @11:54PM
(#36207888)
But for many people training or running inference on large models is important and the apple approach actually makes that feasible in a desktop solution which is pretty awesome.
Two data points from Apple:
Likely 1/4th the size, 1/4th the power consumption, and 4x the ram bandwidth. Have you by chance played with any LLMs? Just saw a post that someone managed 5 tokens/sec with the llama 65B model.