5090: 32GB RAM (newegg & amazon lowest price seems to be +300 more)

4090: 24GB RAM

Thor & Spark: 128GB RAM (probably at least 96GB usable by the GPU if they behave similar to the AMD Strix Halo APU)

True... It would be very interesting to make a comparison of various open models based on token generation speed on these platforms. Presumably starting st some size the larger accessible RAM wins out over raw speed but low VRAM? Although I suppose things like MoE and FP would also matter.

You are doing god's work.

In fact you're also doing the work Nvidia should have done when they put together their (imho) ridiculously imprecise spec sheet.

where does an RTX Pro 6000 Blackwell fall in this? I feel like that’s the next step up in performance (and about the same price as two Sparks)

I thought the 6000 was slightly lower throughput than 5090, but obviously has a shitload more RAM.

It's more throughput, but way less value and there's still no NVLink on the 6000. Something like ~4x the price, ~20% more performance, 3x the VRAM.

There's two models that go by 6000, the RTX Pro 6000 (Blackwell) is the one that's currently relevant.

the RTX Pro 6000 (Blackwell) does not have NVlink? if so, what the fuck Mr.leather jacket.

L40/L40S didn't have it either, which was announced late and felt like a gut-punch for the rare actual non-AI use-case at the time.

A40 nvlink was limited though, one-to-one (bridge), never saw it daisy-chained or nvswitched (might have just not seen them myself, they may have existed).

Of course it doesn't; artificial segmentation because they really want you to buy their even more expensive datacenter GPUs for AI training.

It's not good value when you put it like that. It doesn't have a lot of compute and bandwidth. What it has is the ability to run DGX software for CUDA devs I guess. Not a great inference machine either.

It's great at one thing: memory. And that's interesting because memory is a commodity, but they still make bank on just being able to access it.

Memory is a commodity but access to high bandwidth memory is expensive whether it's HBM, or LPDDR/DDR connected to many memory channels.

Its not priced in a linear way wrt to bom cost.

That's because memory channels cost money. Memory controllers are more complex. Lastly, chips that can make use of high bandwidth VRAM are both of the above.

Memory chips are a commodity, that I agree. Though HBM is trending towards not being a commodity.

Memory controllers are die area. mm2 die space is linear bom cost.

DRAM+mm2 bom will have a different slope to just DRAM bom but still basically linear. Nonlinear pricing is pure market segmentation.

  Memory controllers are die area. mm2 die space is linear bom cost.

Ignoring the design and platform support that comes with higher bandwidth memory controllers.

And this is not also linear or fixed cost, why?

I don't think you know how industry pricing works. Wafers have a price, double mm2, double the price of chip in bom.

Yes. Bigger die size. More complex memory controller designs. More memory lanes. More software support for higher memory lanes. All adds up.

I'm explaining why despite memory being a commodity, high memory bandwidth VRAM cost is not cheap.

Memory controllers are copy paste. It's not more complex.

It's more expensive in a linear way wrt bom.

The RTX Pro 6000 Blackwell gets you 96GB of RAM, a LOT more compute, and costs ~$7K which is not that much more than the $3K-$4K you'd pay for a DGX Spark.

I think the RTX Pro is probably the best deal right now if you're looking for a GPU dev desktop and don't care about physical size or power consumption.

What I ended up doing is buy 3090's in bulk from ex mining rigs, stick new fans in them and run 14 of them connected to an old Supermicro chassis using very nice PCI express splitters. I figured out just what the sweet spot was in terms of PCI bandwidth and maxed it out on RAM (512G). The end result was pretty usable and allowed me to run some of my own benchmarks without sharing any data with the usual suspects. The bigger problem was power delivery, the box had four pretty beefy power supplies running on two different phases. It's retired now because we're much too busy with other stuff but that was a lot of fun and gave me a much better angle on all these new developments than I would have gotten otherwise. Total cost of the system was less than one RTX Pro 6000 and it had a lot more VRAM. The whole thing looked terrible though :)

How does the process management comparison work for GPU vs full systems?

Once the updated Mac Studio with M4/M5 Ultra comes out, pretty much going to make the DGX irrelevant right?

Ultras are pretty expensive.

I mean the spark is $3,999 and current M3 Max 28-Core CPU 60-Core GPU is the same price. I would expect the refreshed studio will stay around the same price.

In Germany the 96gb version is 5000 EUR and the 256gb version is 7000 EUR (no 128gb available as far as I can see).

At that point it's far superior to fly to the US, buy it, and fly back. Hell, have a nice week in a hotel and bring two.

Fly to a state with no sales tax. Portland, Oregon serves this purpose for high end shoppers that come from out of state and out of the country. Folks fly in to buy their Rolex, Gucci, etc, with no tax.

Doesn't work with Germany, you'll have to pay VAT at the airport.

What are they examining your bags for purchased items? That’s gestapoish.

not nessesarily, but you'll have an issue if they do a controll, or it comes up.

1. Yes they will if they suspect you (age group, clothes, newest phone, certain flights like LAX, LGA) as all custom officers all over the world do. As my bags have been searched every time I've entered the US.

2. You should read up on the Gestapo

US customs won't care if they find new electronics, so they're no problem (they are annoying, with the suitcase thing at port-of-entry, but no problem). As for German customs, I don't know, but: do the initial leave out of Germany over the road where there's checks only in theory and leave from an airport not too far over the border. You can probably get a cheaper flight in the process (e.g. Fly from Basel)

The Germans look for electronics, the Americans look in the electronics and what’s on them.

Fly back through Geneva, and take some basic precautions, such as not taking the box. Done/done.

Are you comparing prices with or without taxes? US usually prices without and EU with.

If that would be true why aren't Mac sales banned in China instead of Nvidia GPUs?

Those are high end GPUs that aren’t comparable

Because Tim bribed Trump with a golden calf, or more seriously it's easier to ban a component and its manufacturer vs broader systems.

Not unprecedented though, Playstation 2 had export restrictions.

But it was a different time. Most policies had some connection to the subject at hand.

Policies today are all about brand Trump and brand MAGA.

funny to hear the tards that haven’t listened to any of the policy screeches pretend like they know what’s going on.

Because it's only a superior solution if you just want one box, and that mostly for inference. Once you start scaling to larger loads, it's much trickier to get a clusters of Macs to efficiently process them in parallel, whereas datacenter GPUs are designed for clusters.

I run 4 Mac Studio ultras at work (they’re pricy when maxed out), for local-first AI dev services. But there’s a few things that make me want to switch to the Spark. Networking is the biggest one, the Macs have Thunderbolt and Ethernet, but if I run distributed inference with EXO over Thunderbolt; the drop in tokens/second is massive. These Sparks get RDMA and can stack nicely. The other big one is access to CUDA, MLX has come a long way but being able to have CUDA and GPU access in containers would simplify the stack so nicely. If I had a USB-C/Thunderbolt backplane it might compare, but scaling with the Spark is likely a lot more straightforward.

I call the stack with Mac Studios “MacAIver” because it feels like a duct tape solution, but the Spark equivalent would likely be more elegant.

You'd have to stack 16 of these to get 2TB of VRAM, equivalent to 4 Mac Studios 512GBs chained together.

16 compared to 4. Surely even much faster networking in the Spark would degrade with that many devices?

Biggest problem with Macs is that they don't have dedicated tensor cores in the GPU which makes prompt processing very slow compared to Nvidia and AMD.

n.b. there's been a little speculation that Apple adding TensorOps to Metal 4 suggests M5/M6 may get tensor cores.

https://x.com/liuliu/status/1932158994698932505

https://developer.apple.com/metal/Metal-Shading-Language-Spe...

Nice. I hope so. That would make Macs the best local LLM machines for the masses by far.

It’s $12k for each Mac Studio, and the networking makes them only effective individually (it’s like less that 15 tokens/s with EXO) while NVLINK is very effective. The Spark is definitely more scalable, but the MLX and metal teams are cooking, so honestly either way is still winning.

Note you cannot actually get a 5090 for $1999 that's just the RRP. I believe they actually cost $4k

I just googled it and the first result was one in stock for £2200. That's including tax. I assume $1999 is excluding tax. Without tax and converted to dollars it's $2470.

From other less reliable sources like eBay they are more like £1800.

Huh, you are right. I googled it yesterday too but I guess I had confirmation bias and happened to stumble across an old price and go "yep still 4k".

Well, I'm glad to be wrong on his!

So long as that's true, it's also likely you'll see the same markup for the spark, so they should compare similarly.

The prices came down to near MSRP in the last month or so.

As long as you're going to add FP8 dense, you could do the same for the parts mentioned in the FP4 section. Divide by two from dense => sparse, and another two for FP4 => FP8.

That gives you 250 tops of fp8 for Spark.

Memory is the bottleneck. It limits the size of the models you can run and what you pay for.

  Spark: 128 GB LPDDR5x, unified system memory
  5090 :  32 GB GDDR7,

Model sizes (parameter size)

  Spark: 200B 
  5090 :  12B (raw)

That's very true and what's segmenting the market, but I don't understand why you're saying the 5090 supports only 12B model when it can go up to 50-60B (= a bit less than 64B to leave room for inference) as it supports FP4 as well.

Its for comparison using raw, non optimized models. Both can do much better when you optimize for inference.

Information is in the ratio of these numbers. They stay the same.

Ok then just to clarify: you can fit 4x larger models on the Spark vs 5090, not 17x.

@nabla9 have tried to tell you that for DGX Spark, you can also use optimized models; therefore, this means that Spark can also be used for inference with bigger models, such as those exceeding 200B.

Please compare the same things: carrots VS carrots, not apples VS eggs.

I don't understand what's not optimized on 5090. If we're comparing with Apple chips or AMD Strix Halo yes you will have very different hardware + software support, no FP4 etc. but here everything is CUDA, Blackwell vs Blackwell, same FP4 structured sparsity, so I don't get how it would be honest to compare a quantized FP4 model on Spark with an unoptimized FP16 model on a 5090 ?

To me, what I think they are saying is that the Spark can use a FP16 unoptimized model with 200B parameters. However I don't really know.

You can't. The Spark has 128GB VRAM; the highest you can go in FP16 is 64B — and that's with no space for context.

200B is probably a rough estimate of Q4 + some space for context.

The Spark has 4x the VRAM of a 5090. That's all you need to know from a "how big can it go" perspective.

from the NVidia DGX Spark datasheet:

  With 128 GB of unified system memory, developers can experiment, fine-tune, or inference models of up to 200B parameters. Plus, NVIDIA ConnectX™ networking can connect two NVIDIA DGX Spark supercomputers to enable inference on models up to 405B parameters.

The datasheet isn't telling you the quantization (intentionally). Model weights at FP16 are roughly 2GB per billion params. A 200B model at FP16 would take 400GB just to load the weights; a single DGX Spark has 128GB. Even two networked together couldn't do it at FP16.

You can do it, if you quantize to FP4 — and Nvidia's special variant of FP4, NVFP4, isn't too bad (and it's optimized on Blackwell). Some models are even trained at FP4 these days, like the gpt-oss models. But gigabytes are gigabytes, and you can't squeeze 400GB of FP16 weights into only 128GB (or 256GB) of space.

The datasheet is telling you the truth: you can fit a 200B model. But it's not saying you can do that at FP16 — because you can't. You can only do it at FP4.

I never claimed the 200B model was FP16.

If the 200B model was at FP16, marketing could've turned around and claimed the DGX Spark could handle a 400B model (with an 8-bit quant) or a 800B model at some 4-bit quant.

Why would marketing leave such low-hanging fruit on the tree?

They wouldn't.

You and nabla9 are both the one comparing apples and eggs. 4x more RAM means 4x larger models when everything else is held the same to make a fair comparison.

So how many generation T/s we can expect for a dense model?

I assume we can go up to 120B using fp8?