Fully agree. It's time for some serious competition not just against Intel/x86 but also in the ARM space.

IDK about ML part, but equivalent performing Ryzen mini pc cost me 3x less over m1 macbook (yes im aware you get more with macbook)

It's easy to outperform Apple Silicon on pure power, but what about efficiency & heat? (like FLOPS/watt or whatever). Does anything else come close yet?

That matters a lot in a laptop, but not so much in a 1U rack? Not that datacenters love heat, but the competition isn't extra hot, it's just hotter than Apple Silicon?

I guess it depends on what you're doing with them? If you're running them 24/7 to train or model something, the energy costs might add up. Even if you're not, having more efficient chips might mean more data centers don't need as complex cooling equipment.

A 1U rack is a small machine, but a few hundred of them+ in a constrained space is a different story. At larger scales, moving electricity in and heat out are usually the defining factors.

You're quite right - 6800H TDP is 45W, M1 is 30W

> IDK about ML part, but equivalent performing Ryzen mini pc cost me 3x less over m1 macbook

When running an ML workload, the Nvidea A100 has massive GPU compute resources, and a large amount of GPU local high bandwidth memory, so it's ideal, but is nowhere near low cost.

A consumer Ryzen chip is inexpensive, but lacks in both memory bandwidth and GPU resources.

The M2 Ultra has access to way more RAM than consumer GPUs, many times the memory bandwidth of a Ryzen (800GB/s vs Ryzen 7 1800X at 40 GB/s) with a large amount of local GPU resources.

Even stepping up to a Threadripper Pro would only get you a quarter of the memory bandwidth, and those aren't exactly cheap either.

There are multiple riscv based solutions that will be out in 12-18 months. But for now, getting Apple hardware is the best solution.

you can buy rack mount Mac pro

Apple silicon is great for low power desktops and laptops, but they don't actually have groundbreaking performance relative to what we've got in the server space. If you dropped the M2 Ultra from the $4000 into a server, it would perform about the same as a $1500 AMD 7950X3D based server (this is a common budget server setup with ECC) in CPU tasks. Stick a common GPU in there and you're running circles around the M2 Silicon in GPU tasks.

The Apple Silicon is great at really low power work, but if you dial desktop or server GPU power limits down they also become quite efficient. The marginal cost of electricity is cheaper than buying more hardware, so nVidia and others run their parts deep into the diminishing returns part of the curve to maximize performance at the expense of power efficiency.

What Apple Silicon brings to the table is not simply just performance, but a large amount of unified memory that can be used by the GPU (which are needed for inference of large deep neural networks like LLMs).

A top-of-the-line Mac Studio will give you 192 GB of unified RAM in less than $7000. Meanwhile a H100 from NVIDIA with 80 GB of VRAM will cost you like $30000...

If the software were a little better Apple Silicon would be by far the best rig for DIY LLM research or training.

192GiB RAM is enough to train or inference Falcon 180B in RAM at 8-bit resolution.