A single 3090 will deliver more tflops than the m2 ultra.

The 3090s also give you access to all those CUDA inference and training frameworks used in industry. With Apple you're pretty much stuck with either MLX or llama.cpp.

Honestly though I'd be curious to see a cost analysis of Apple vs. Nvidia for commercial batched inference. The Nvidia system can obviously spit out more tokens/s but for the same price you could have multiple Mac Studios running the same model (and users would be dispatched to one of them).

The M2 Ultra doesn't require doing electrical work on your house like this 8x 3090 setup did though.

1x 3090 would be faster than the mac, cheaper than the mac, and run on a normal residential circuit. How many breakers would you need for a cluster of 20 macs running at full throttle to match the perf of OP's build?

A single 3090 won’t even fit the model. OP is talking about running a 405B model, it needs close to 200gb of memory just to open it which is why we’re talking about mac studio and it’s 192gb unified memory.

This is exactly like when the AMD fanboys got a burr up their ass about the “$50k Mac Pro” with 2tb of memory… when you could the same thing with a threadripper with 256gb of memory for $5k, and it’s just as fast in Cinebench!

https://old.reddit.com/r/Amd/comments/f1a0qp/15000_mac_pro_v...

your gaming scores on your 3090 with 24gb are just as irrelevant to this 200gb workload as the threadripper is to those Mac Pro workloads lol

You could power limit the 3090s to fit a standard 120V*20A = 2400W outlet if you really want to. The default power limit is 350W each so you'll only lose a little perf.

Also most rooms have multiple circuits. Just connect half the GPUs to each outlet. I already do this with my desktop PC because it has 2 PSUs.

Also most homes in the US have 30A*240V = 7200W dryer/stove outlets in the kitchen, laundry room, garage, etc. That's where you want to keep loud computers.

Brb replacing my stove with a bunch of GPUs

Easy-bake cryptominer

Inference Oven

You can cook on the GPUs from now on.

A standard outlet is 15A, and you're only allowed to use 80% continuously. So now you're dropping to 150W each.

Using two circuits is viable.

In the bay area every home I've lived in had 20A circuits for everything.

The NEC 80% rule is only relevant in data centers where someone enforces it. The time-current curve for any 20A breaker shows that it will take infinite time to trip when you draw 20A. This is correct behavior because 12awg wire can handle 20A indefinitely

Do they actually install the 20A outlets with the sideways part? I don't know if I've ever seen one in person.

I guess the outlet doesn't matter a whole lot, especially if you take multiple cables to the wall.

But I still don't want to be doing unenforced electrical violations.

Some outlets have the 20A sideways part, but from a safety/code perspective that's irrelevant.

Consider what happens when you plug two 10A devices into a power strip. In the US there are no true 15A rated outlets. All outlets must handle 20A because the electrical code allows any outlet to be on a 20A breaker.

> Consider what happens when you plug two 10A devices into a power strip.

I don't know about any argument that depends on such an action being safe. There's plenty of power strips I wouldn't trust with 20 amps.

That's just how the US electrical code works. I assure you that every "15A" outlet on the wall can handle 20A. That's why products like this don't have a fuse: https://www.homedepot.com/p/Woods-2-ft-12-3-SJTW-Multi-Outle...

Location specific issue. Higher voltage resolves this and a regular household socket will power this just fine. Then again, if you're spending that much money to get the job done, new wiring is probably one of the cheaper parts of the build.

You can run 8 or even more 3090s without doing electrical work on a standard US residential power feed, you just need outlets connected to multiple circuits. This is not always convenient, which is why some may choose to do electrical work, but it’s absolutely not required.

Yes, for 20x the FLOPS at about twice the price you might need to do some work.

Sure, neither does just using ChatGPT, but we don't do things because they are easy, we do them because they are difficult.

Yeah cause it's doing like 50x less work.

Yep. The 8x3090 setup should be at least 10 times faster than the Mac.

Yes though for inference memory bandwidth is more important than tflops. Not sure how Apple compares in that regard. The OP does train models too though which is more compute heavy.

Each of the 3090s has something like 15% higher memory bandwidth than the whole M2 Ultra system does.

He's got 8x3090s are you fucking kidding? Like is this some kind of AI reply?

"Wow great post! I enjoy your valuable contributions. Can you tell me more about graphics cards and how they compare to other different types of computers? I am interested and eager to learn! :)"

It's one thing to be an asshole, but you're also hilariously clueless.

Yeah, because an M2 is in the same ballpark as 8 GPUs. Yes, you can use CPU now but it's not even close to this setup. This is hackernews. I know we're supposed to be nice and this isn't reddit, but comments like parent are ridiculous and for sure don't add to the discussion any more than mine do.

I simply didn’t know the answer and the response “it would be much slower” is a perfectly acceptable reply. I disagree that it was ridiculous to ask. I was curious and I wanted to know the answer and now I do. What is obvious to you is not obvious to other people, and you will get nowhere in life insulting people who are asking questions in good faith.

What? No I just don’t know the difference, sorry. I am interested in learning more about running 405b parameter models, which I believe you can do on a 192gb M series Mac.

The answer here is that the Nvidia system has much better performance. I’ve been focused on “can I even run the model” I didn’t think about the actual performance of the system.

You're interested in the different between a single CPU and 8 GPUs? A Ford fiesta vs a freight train.

One can be interested in the differences between a Ford Fiesta and a freight train…

Are you fucking kidding me? A single train car can weight 130 tons, a fiesta can carry maybe 500kg, its not even close. /s

How is that even sarcasm

A single SoC, which includes a GPU (two GPUs, kinda).

Yeah. I can’t afford a freight train.

Keep an eye on the SV going out of business fire sales. Not all the AI kites will fly.

As for actual trains, they can be suprisingly affordable (to live in):

https://atrservices.com.au/product/sa-red-hen-416/ https://en.wikipedia.org/wiki/South_Australian_Railways_Redh...

and the freight rolling stock flatcars make great bridges (single or sectioned) with concrete pylons either end for farms - once the axles are shot they can go pretty damn cheap and the beds are good enough to roll a car or small truck over.

addendum: in case you miss fresh reply to old comment: https://news.ycombinator.com/item?id=41484529

It's kinda hard to believe that someone would stumble onto the landmine of AI performance comparison between Apple Silicon and Nvidia hardware. People are going to be rude because this kinda behavior is genuinely indistinguishable from bad-faith trolling. From benchmarks alone, you can easily tell that the performance-per-watt of any Mac Studio gets annihilated by a 4090: https://browser.geekbench.com/opencl-benchmarks

If Apple Silicon was in any way a more scalable, better-supported or more ubiquitous solution, then OpenAI and the rest of the research community would use their hardware instead of Nvidia's. Given Apple's very public denouncement of OpenCL and the consequences of them refusing to sign Nvidia drivers, Apple's falling-behind in AI is like the #1 topic in the tech sector right now. Apple Silicon for AI training is a waste of time and a headache that is beyond the capacity of professional and productive teams. Apple Silicon for AI inference is too slow to compete against the datacenter incumbents fielded by Nvidia and even AMD. Until Apple changes things and takes the datacenter market seriously (and not just advertise that they are), this status quo will remain the same. Datacenters don't want to pay the Apple premium just so they can be treated like a traitorous sideshow.

> From benchmarks alone, you can easily tell that the performance-per-watt of any Mac Studio gets annihilated by a 4090: https://browser.geekbench.com/opencl-benchmarks

The Geekbench GPU compute benchmarks are nearly worthless in any context, and most certainly are useless for evaluating suitability for running LLMs, or anything involving multiple GPUs.

> It's kinda hard to believe that someone would stumble onto the landmine of AI performance comparison between Apple Silicon and Nvidia hardware.

I encourage you to update your beliefs about other people. I’m a very technical person, but I work in robotics closer to the hardware level - I design motor controllers and Linux motherboards and write firmware and platform level robotics stacks, but I’ve never done any work that required running inference in a professional capacity. I’ve played with machine learning, even collecting and hand labeling my own dataset and training a semantic segmentation network. But I’ve only ever had my little desktop with one Nvidia card to run it all. Back in the day, performance of CNNs was very important and I might have looked at benchmarks, but since the dawn of LLMs, my ability to run networks has been limited entirely by RAM constraints, not other factors like tokens per second. So when I heard that MacBooks have shared memory and can run large models with it, I started to notice that could be a (relatively) accessible way to run larger models. I can’t even remotely afford a $6k Mac any more than I could afford a $12k Nvidia cluster machine, so I never really got to the practical considerations of whether there would be any serious performance concerns. It has been idle thinking like “hmm I wonder how well that would work”.

So I asked the question. I said roughly “hey can someone explain why OP didn’t go with this cheaper solution”. The very simple answer is that it would be much slower and the performance per dollar would be 10x worse. Great! Question answered. All this rude incredulousness coming from people who cannot fathom that another person might not know the answer is really odd to me. I simply never even thought to check benchmarks because it was never a real consideration for me to buy a system.

Also the “#1 topic in the tech sector right now” funny in my circles people are talking about unions, AI compute exacerbating climate change, and AI being used to disenfranchise and make more precarious the tech working class. We all live in bubbles.

It's simply bizarre that you would ask that question when the research to figure it all out is trivially accessed. Everyone thought that "unified memory" would be a boon when it was advertised, but Apple never delivered on a CUDA alternative. They killed OpenCL in the cradle, and pushed developers to use Metal Compute Shaders instead of a proper GPGPU layer. If you are an Apple dev, the mere existence of CoreML ought to be the white flag that makes you realize Apple hardware was never made for GPU compute.

Again, I'm not accusing you of bad-faith. I'm just saying that asking such a bald-faced and easily-Googled question is indistinguishable from flamebait. There is so much signalling that should suggest to you that Apple hardware is far from optimized for AI workloads. You can look at it from the software angle, where Apple has no accessible GPGPU primitives. You can look at it from a hardware perspective, where Apple cannot beat the performance-per-watt of desktop or datacenter Nvidia hardware. You can look at it from a practical perspective, where literally nobody is using Apple Silicon for cost-effective inference or training. Every single scrap of salient evidence suggests that Apple just doesn't care about AI and the industry cannot be bothered to do Apple's dirty work for them. Hell, even a passing familiarity with the existence of Xserve should say everything you need to know about Apple competing in markets they can't manipulate.

> funny in my circles people are talking about unions, AI compute exacerbating climate change, and AI being used to disenfranchise and make more precarious the tech working class.

Sounds like your circles aren't focused on technology, but popular culture and Twitter topics. Unionization, the "cost" of cloud and fictional AI-dominated futures were barely cutting-edge in the 90s, let alone today.

You might enjoy last fortnight's The Register article:

Buying a PC for local AI? These are the specs that matter

https://www.theregister.com/2024/08/25/ai_pc_buying_guide/

It died without interest here: https://news.ycombinator.com/item?id=41347785 likely time of day, I'm mostly active in HN off peak hours.

Nice, thank you!

While this reply might be a bit too harsh, I fully agree with the well warranted criticism of Apple fans chiming in on every AI Nvidia discussion with “but M chips have large amount of RAM and Apple says they’re amazing for AI”.

kind of violent approach but I agree on the bottom line. I don't see why should somebody be enthousiast about this. someone was just able to spend 8× the figures a random teen is able to spend for his gaming rig, and he just iterated the teen's rig 8 times, he then installed ubuntu+cuda and called it a day.

something that is actually interesting that is attempting to bring something on the table : check tinygrad/tinycorp