News for hackers. Stuff that matters.
Re:Apple: Transformer architecture optimized for Apple Silicon by api on Thursday March 23, 2023 @07:18PM (#35282878)
Apple M series chips are already some of the best chips for running your own models locally, especially if you want a laptop or small form factor machine instead of a big gaming desktop with a big GPU.

Apple is really getting a niche here for machines to run models locally. That’s pretty powerful.

Re:Apple: Transformer architecture optimized for Apple Silicon by smoldesu on Thursday March 23, 2023 @07:50PM (#35283168)
> Apple is really getting a niche here for machines to run models locally.

I... don't agree. They have an acceleration API and a large install-base, but all of these models have run just fine on traditional hardware. GPT-Neo, Stable Diffusion and LLaMa can all be used and accelerated without Apple Silicon.

Powerful, maybe. But not really unique, just putting up table stakes.

Re:Apple: Transformer architecture optimized for Apple Silicon by itg on Thursday March 23, 2023 @08:10PM (#35283340)
I would not say they are the best chips if you want to run locally. They are ok although a bit slow compared to desktop gpus for inference.
Re:Apple: Transformer architecture optimized for Apple Silicon by jazzyjackson on Thursday March 23, 2023 @09:06PM (#35283744)
What desktop GPUs would you recommend for the home user with access to 20+ GB of VRAM? I can get a Mac Studio with 128GB of shared RAM for about 5k.
Re:Apple: Transformer architecture optimized for Apple Silicon by lostmsu on Friday March 24, 2023 @01:46AM (#35285476)
What's the point of shared RAM exactly? 2x3090 ($2000) + 128GB RAM ($300) to cache weights would be much faster. Probably around 10-20x.
Re:Apple: Transformer architecture optimized for Apple Silicon by dragonwriter on Friday March 24, 2023 @03:31AM (#35286120)
> What desktop GPUs would you recommend for the home user with access to 20+ GB of VRAM?

I think the usual current model would be the 24GB Nvidia RTX 4090.