News for hackers. Stuff that matters.
Re:Apple: Transformer architecture optimized for Apple Silicon by brookst on Friday March 24, 2023 @09:45AM (#35289001)
ML inference is not generally FP32 anymore. I was going off of the TOPs numbers for ML from a few sources, which generally agree M2 is about 22 TOPS and 4080 (desktop) is about 50.

But in any event, yes, that was my point. UMA is a huge advantage, the GPU itself is too weak to be serious.

But it’s a lot easier to drop a dramatically beefier GPU into a new design than it is to update the entire platform for UMA. Apple has a huge opportunity here… whether tbey pursue it or not remains to be seen.

Re:Apple: Transformer architecture optimized for Apple Silicon by smoldesu on Friday March 24, 2023 @10:50AM (#35289773)
> whether tbey pursue it or not remains to be seen.

Pursue what though?

UMA is cool, but kinda meaningless if the majority of Macbooks are min-spec. That leaves you with 4-5gb of VRAM, assuming you've left nothing open. What is Apple going to do with that UMA that other manufacturers cannot?

It's certainly nice that 128gb Macs exist for models that might be too big to otherwise load into memory. It's useless for production inferencing though, and I struggle to imagine the "opportunities" they're missing out on here.

Re:Apple: Transformer architecture optimized for Apple Silicon by brookst on Friday March 24, 2023 @02:47PM (#35293747)
> Pursue what though?

A Mac variant that trades CPU cores for GPU/ML cores while having 192GB+ of UMA memory.

> I struggle to imagine the “opportunities”

Two of them: 1) academic / R&D compute, where people could have at least A6000 class GPU on the desktop, and 2) cloud inference servers, probably for Apple’s own services.

I’m not saying they will or should do those things, just that the apple silicon arch is well positioned if they choose to. Bolting on exponentially better GPU is not especially difficult, and they’ve got an OS that would bring existing apps and libraries right over.

Look at it this way: is there a path to UMA on Windows / Linux? If not, those systems will always duplicate RAM and require users to decide in advance whether to allocate RAM budget to OS or ML.

Re:Apple: Transformer architecture optimized for Apple Silicon by smoldesu on Friday March 24, 2023 @03:20PM (#35294144)
Academics might buy in, but they're a small market and still fairly easy to poach with quality server hardware. You may be right about Apple using them for cloud inferencing though, seeing how they'd rather pound sand than patch up their relationship with Nvidia.

Whichever way you look at it though, neither of those are really opportunities. Apple boxed themselves into the consumer market, and now has to compete with professionally-priced products.

Re:Apple: Transformer architecture optimized for Apple Silicon by brookst on Friday March 24, 2023 @10:43PM (#35298720)
I think you vastly overestimate the emotionality of corporate execs.