Apple's On-Device and Server Foundation Models

Re:Apple's On-Device and Server Foundation Models by mrweasel on Tuesday June 11, 2024 @03:06AM (#40643413)

The interesting point here is that developers targeting the Mac can safely assume that the users will have a processor capable of significant AI/ML workloads. On the Windows (and Linux) side of things, there's no common platform, no assumption that the users will have an NPU or GPU capable of doing what you want. I think that's also why Microsoft was initially going for the ARM laptops, where they'd be sure that the required processing power is available.

[ Parent | on HN ]

Re:Apple's On-Device and Server Foundation Models by skohan on Tuesday June 11, 2024 @04:06AM (#40643759)

I believe MS is trying to standardize this, in the same way as they do with DirectX support levels, but I agree it's probably going to be inherently a bit less consistent than Apple offerings

[ Parent | on HN ]

Re:Apple's On-Device and Server Foundation Models by pjmlp on Tuesday June 11, 2024 @07:22AM (#40644950)

DirectML can use multiple backends.

[ Parent | on HN ]

Re:Apple's On-Device and Server Foundation Models by qwytw on Tuesday June 11, 2024 @03:19AM (#40643505)

> The interesting point here is that developers targeting the Mac can safely assume that the users will have a processor capable of significant AI/ML workloads

Also that a significant proportion (majority?) of them will have just 8 GB of memory which is not exactly sufficient to run any complex AI/ML workloads.

[ Parent | on HN ]

Re:Apple's On-Device and Server Foundation Models by talldayo on Tuesday June 11, 2024 @03:10PM (#40650211)

Easy solution; just swap multiple gigabytes of your model to SSD-based ZRAM when you run out of memory. What could possibly go wrong?

[ Parent | on HN ]

Re:Apple's On-Device and Server Foundation Models by sofixa on Tuesday June 11, 2024 @03:55AM (#40643703)

That's probably where Microsoft's "Copilot+ PCs" come in.

[ Parent | on HN ]

Re:Apple's On-Device and Server Foundation Models by pjmlp on Tuesday June 11, 2024 @07:23AM (#40644955)

Plus DirectML, wich as the name implies, builds on top of DirectX, allowing multiple backends, CPU, GPU, NPU.

[ Parent | on HN ]

Re:Apple's On-Device and Server Foundation Models by treprinum on Tuesday June 11, 2024 @04:23AM (#40643853)

How does it help me (with maxed out M3 Max) that Apple might have some chip in the future right now? I do DL on A6000 and 4090, not waiting until Apple produces a chip someday that is faster than 1650 in ML...

[ Parent | on HN ]

Re:Apple's On-Device and Server Foundation Models by InsomniacL on Tuesday June 11, 2024 @05:29AM (#40644225)

That sounds like a big issue, but surely assuming for either case is bad.

I expect OS's will expose an API which, when queried, will indicate the level of AI inference available.

Similar to video decoding/encoding where clients can check if hardware acceleration is available.

[ Parent | on HN ]