Which is neat, but it's not CUDA. It's an application-specific accelerator good at a small subset of operations, controlled by a high-level library the industry is unfamiliar with and too underpowered to run LLMs or image generators. The NPU is a novelty, and today's presentation more-or-less confirmed how useless it is for rich local-only operations.
> Apple also has the means to build data center training hardware using apple silicon if they want to do so.
They could, but that's not a competitor against an NVL72 with hundreds of terabytes of unified GPU memory. And then they would need a CUDA competitor, which could either mean reviving OpenCL's rotting corpse, adopting Tensorflow/Pytorch like a sane and well-reasoned company, or reinventing the wheel with an extra library/Accelerate Framework/MPS solution that nobody knows about and has to convert models to use.
So they can make servers, but Xserve showed us pretty clearly that you can lead a sysadmin to MacOS but you can't make them use it.
> they could also start to supply them with cloud inference hardware and strongarm them into only using apple servers to serve iOS requests.
I wonder how much money they would lose doing that, over just using the industry-standard Nvidia servers. Once you factor in the margins they would have made selling those chips as consumer systems, it's probably in the tens-of-millions.
This is Apple's favorite thing in the world. They already have an Apple-Silicon-only ML framework as of a few months ago, called MLX. Does anyone know about it? No. Do you need to convert models to use it? Yes.
Users absolutely don't care if their prompt response has been generated by a CUDA kernel or some poorly documented apple specific silicon a poor team at cupertino almost lost their sanity to while porting the model.
And haven't they already spent quite a bit on money on their pytorch-like MLX framework?
They most certainly will. If you run GPT-4o on an iPhone with MLX, it will suck. Users will tell you it sucks, and they won't do so in developer-specific terms.
The entire point of this thread is that Apple can't make users happy with their Neural Engine. They require a stopgap cloud solution to make up for the lack of local power on iPhone.
> And haven't they already spent quite a bit on money on their pytorch-like MLX framework?
As well as Accelerate Framework, Metal Performance Shaders and previously, OpenCL. Apple can't decide where to focus their efforts, least of which in a way that threatens CUDA as a platform.
But the point stands, these systems occupy a niche that Apple Silicon is poorly suited to filling. They run normal Linux, they support common APIs, and network to dozens of other machines using Infiniband.
Inference can run without it, and could so for years via ONNX. Now we are starting to see more back-ends becoming available.
> If they are serious about the OAI partnership they could also start to supply them with cloud inference hardware and strongarm them into only using apple servers to serve iOS requests
Apple addressed both these points in today’s preso.
1. They will send requests that require larger contexts to their own Apple Silicon-based servers that will provide Apple devices a new product platform called Private Cloud Compute.
2. Apple’s OS generative AI request APIs won’t even talk to cloud compute resources that do not attest to infrastructure that has a publicly available privacy audit.
You’re absolutely right. I got too excited about Apple’s strategy to encourage developers to use Apple Private Cloud Compute.
The UX for ChatGPT as shown for iOS 18 makes it obvious that you are sending data outside the Apple Silicon walled garden.
i'm seeing people all over this thread saying stuff like that, it reads like fantasyland to me. Apple doesn't have the talent or the chips or suppliers or really any of the capabilities to do this, where are people getting it from?
Their ability to design a chip and networking fabric which is fast/efficient at training a narrow set of model architecture is not far fetched by any means.
They are the first ones to ship on-device inference at scale on non-nvidia hardware. Apple also has the means to build data center training hardware using apple silicon if they want to do so.
If they are serious about the OAI partnership they could also start to supply them with cloud inference hardware and strongarm them into only using apple servers to serve iOS requests.