Can you actually use multiple videocards easily with existing AI model tools?

Yes, though how you do it depends on what you're doing.

I do a lot of training of encoders, multimodal, and vision models, which are typically small enough to fit on a single GPU; multiple GPUs enables data parallelism, where the data is spread to an independent copy of each model.

Occasionally fine-tuning large models and need to use model-parallelism, where the model is split across GPUs. This is also necessary for inference of the really big models, as well.

But most tooling for training/inference of all kinds of models supports using multiple cards pretty easily.

Yes, multi-GPU on the same machine is pretty straightforward. For example ollama uses all GPUs out of the box. If you are into training, the huggingface ecosystem supports it and you can always go the manual route to put tensors on their own GPUs with toolkits like pytorch.

Yes. Depends what software you're using. Some will use more than one (e.g. llama.cpp), some commercial software won't bother.

I just made a video on this very thing: https://youtu.be/JtbyA94gffc

Sounds like you might be more the target for the $3k 128GB DIGITS machine.

Weirdly they're advertising "1 petaflop of AI performance at FP4 precision" [1] when they're advertising the 5090 [2] as having 3352 "AI TOPS" (presumably equivalent to "3 petaflops at FP4 precision"). The closest graphics card they're selling is the 5070 with a GPU performing at 988 "AI TOPS" [2]....

[1] https://nvidianews.nvidia.com/news/nvidia-puts-grace-blackwe...

[2] https://www.nvidia.com/en-us/geforce/graphics-cards/50-serie...

It seems like they're entirely different units?

TOPS means Tera Operations Per Second.

Petaflops means Peta Floating Point Operations Per Second.

... which, at least to my uneducated mind here, doesn't sound comparable.

I'm assuming that the operations being referred to in both cases are fp4 floating point operations. Mostly because

1. That's used for AI, so it's plausibly what they mean by "AI OPS"

2. It's generally a safe bet that the marketing numbers NVIDIA gives you is going to be for the fastest operations on the computers, and that those are the same for both computers when they're based on the same architecture.

Other than that, Terra is 10^12, Peta is 10^15, so 3352 Tera ops is 3.352 Peta ops and so on.

I’m really curious what training is going to be like on it, though. If it’s good, then absolutely! :)

But it seems more aimed at inference from what I’ve read?

I was wondering the same thing. Training is much more memory-intensive so the usual low memory of consumer GPUs is a big issue. But with 128GB of unified memory the Digits machine seems promising. I bet there are some other limitations that make training not viable on it.

Primarily concerned about the memory bandwidth for training.

Though I think I've been able to max out my M2 when using the MacBook's integrated memory with MLX, so maybe that won't be an issue.

Training is compute bound, not memory bandwidth bound. That is how Cerebras is able to do training with external DRAM that only has 150GB/sec memory bandwidth.

The architectures really aren't comparable. The Cerebras WSE has fairly low DRAM bandwidth, but it has a huge amount of on-die SRAM.

https://www.hc34.hotchips.org/assets/program/conference/day2...

They are training models that need terabytes of RAM with only 150GB/sec of memory bandwidth. That is compute bound. If you think it is memory bandwidth bound, please explain the algorithms and how they are memory bandwidth bound.

It will only have 1/40 performance of BH200, so really not enough for training.

What I really don't like about it is low power GPUs appear to be a thing of the past essentially. An APU is the closest you'll come to that which is really somewhat unfortunate as the thermal budget for an APU is much tighter than it has to be for a GPU. There is no 75W modern GPU on the market.

the closest is the L4 https://www.nvidia.com/en-us/data-center/l4/ but its a bit weird.

RTX A4000 has an actual display output

Innodisk EGPV-1101

I heavily power limited my 4090. Works great.

Yep. I use ~80% and barely see any perf degradation. I use 270W for my 3090 (out of 350W+).

Performance per watt[1] makes more sense than raw power for most consumer computation tasks today. Would really like to see more focus on energy efficiency going forward.

[1] https://en.wikipedia.org/wiki/Performance_per_watt

That's s blind way to look at that imho. Doesn't work on me for sure.

More energy means more power consumption, more heat in my room, you can't escape thermodynamics. I have a small home office, it's 6 square meters, during summer energy draw in my room makes a gigantic difference in temperature.

I have no intention of drawing more than a total 400w top while gaming and I prefer compromising on lowering settings.

Energy consumption can't keep increasing over and over forever.

I can even understand it on flagships, they meant for enthusiasts, but all the tiers have been ballooning in energy consumption.

Increasing performance per watt means that you can get more performance using the same power. It also means you can budget more power for even better performance if you need it.

In the US the limiting factor is the 15A/20A circuits which will give you at most 2000W. So if the performance is double but it uses only 30% more power, that seems like a worthwhile tradeoff.

But at some point, that ends when you hit a max power that prevents people from running a 200W CPU and other appliances on the same circuit without tripping a breaker.

> Increasing performance per watt means that you can get more performance using the same power.

I'm currently running a 150 watt GPU, and the 5070 has a 250 TDP. You are correct. I could get a 5070 and down volt it to work in 150ish range e.g. and get almost the same performance (at least not significantly different to notice in game).

But I think you're missing the wider point of my complain: it's been from Maxwell that Nvidia hasn't produced major updates on the power consumption side of their architecture.

Simply making bigger and denser chips on better nodes while keeping to increase the power draw and slapping DLSS4 is not really an evolution, it's laziness and milking the users.

On top of that: the performance benefits we're talking about are really using DLSS4, which is artificially limited to the latest gen. I don't expect raw performance of this gen to exceed a 20% bump to the previous one when DLSS is off.

> But I think you're missing the wider point of my complain: it's been from Maxwell that Nvidia hasn't produced major updates on the power consumption side of their architecture.

Is this true or is it just that the default configuration draws a crazy amount of power? I wouldn't imagine running a 5090 downvolted to 75W is useful, but also I would like to see someone test it against an actual 75W card. I've definitely read that you can get 70% of the performance for 50% of the power if you downvolt cards, and it would be interesting to see an analysis of what the sweet spot is for different cards.

Today's hardware typically consumes as much power as it wants, unless we constrain it for heat or maybe battery.

If you're undervolting a GPU because it doesn't have a setting for "efficiency mode" in the driver, that's just kinda sad.

There may be times when you do want the performance over efficiency.

I remember various benchmarks in the years since Maxwell, when they took GPUs with comparable number of cuda cores and clocked them the same, the performance were in the delta of error suggesting that (raw) performance wise you're not getting much more since Maxwell (and that was what, 2013?).

I can confirm you that downvolting can get you the same tier of performance (-10%, which by the way is 3 fps when you're making 3 and 10 when you're making 100, negligible) by cutting power consumption by a lot, how much is that a lot depends on the specific gpu. On the 4090 you can get 90% of the performance at half the power draw, lower tier car have smaller gain/benefits ratios.

I got into desktop gaming at the 970 and the common wisdom (to me at least, maybe I was silly) was I could get away with a lower wattage power supply and use it in future generations cause everything would keep getting more efficient. Hah...

I went from 970 to 3070 and it now draws less power on average. I can even lower the max power to 50% and not notice a difference for most games that I play.

Yeah, do like me, I lower settings from "ultra hardcore" to "high" and keep living fine on a 3060 at 1440p for another few gens.

I'm not buying GPUs that expensive nor energy consuming, no chance.

In any case I think Maxwell/Pascal efficiency won't be seen anymore, with those RT cores you get more energy draw, can't get around that.

I'm generally a 1080p@60hz gamer and my 3060 Ti is overpowered for a lot of the games I play. However, there are an increasing number of titles being released over the past couple of years where even on medium settings the card struggles to keep a consistent 60 fps frame rate.

I've wanted to upgrade but overall I'm more concerned about power consumption than raw total performance and each successive generation of GPUs from nVidia seems to be going the wrong direction.

I think you can get a 5060 and simply down volt it some, you'll get more or less the same performance while reducing power draw sensibly.

That's probably not going to be an option for me as I wanted to upgrade to something with 16 GB of vram. I do toy with running LLM inference and squeezing models to fit in 8 GB vram is painful. Since the 5070 non-ti has 12 GB of vram there is no hope that a 5060 would have more vram than that. So, at a minimum I'm stuck with the prospect of upgrading to a 5070 ti.

That's not the end of the world for me if I move to a 5070 ti and you are quite correct that I can downclock/undervolt to keep a handle on power consumption. The price makes it a bit of a hard pill to swallow though.

I feel similarly; I just picked up a second hand 6600 XT (similar performance to 3060) and I feel like it would be a while before I'd be tempted to upgrade, and certainly not for $500+, much less thousands.

8Gb VRAM isn't enough for newer games though.

I've actually reversed my GPU buying logic from the old days. I used to buy the most powerful bleeding edge GPU I could afford. Now I buy the minimum viable one for the games I play, and only bother to upgrade if a new game requires a higher minimum viable GPU spec. Also I generally favor gameplay over graphics, which makes this strategy viable.

Yeah, that's another fact.

I upgrade GPUs then keep launching League of Legends and other games that really don't need much power :)

I thought opposite. My powersupply is just another component. I'll upgrade it as I need to. But keeping it all quiet and cool...

I built a gaming PC aiming to last 8-10 years. I spent $$$ on MO-RA3 radiator for water cooling loop.

My view:

1. a gaming PC is almost always plugged into a wall powerpoint

2. loudest voices in the market always want "MOAR POWA!!!"

1. + 2. = gaming PC will evolve until it takes up the max wattage a powerpoint can deliver.

For the future: "split system aircon" built into your gaming PC.

For the curious what I actually did was stop gaming and haven't bought a GPU since 2000's! GPU stuff is still interesting to me, though.

I stopped playing a lot of games post-2010/2014 or so.

Lots of games that are fine on Intel Integrated graphics out there.

You don't need to run them in x16 mode though. For inference even half that is good enough.

most household circuits can only support 15-20 amps at the plug. there will be an upper limit to this and i suspect this is nvidia compromising on TDP in the short term to move faster on compute

I wonder if they will start putting lithium batteries in desktops so they can draw higher peak power.

There's a company doing that for stovetops, which I found really interesting (https://www.impulselabs.com)!

Unfortunately, when training on a desktop it's _relatively_ continuous power draw, and can go on for days. :/

Yeah that stove is what I was thinking of!

And good point on training. I don't know what use cases would be supported by a battery, but there's a marketable one I am sure we will hear about it.

They already use capacitors for that.

Batteries and capacitors would serve different functions. Capacitors primarily isolate each individual chip and subsystem on a PCB from high frequency power fluctuations when digital circuits switch or larger loads turn on or off. You would still need to use capacitors for that. The purpose of the batteries would be to support high loads on the order of minutes that exceed the actual wall plug capacity to deliver electricity. I am thinking specifically of the stove linked in your sibling comment, which uses lithium batteries to provide sustained bursts of power to boil a pot of water in tens of seconds without exceeding the power ratings of the wall plug.

It is the same function on different time scales. If you had a big enough capacitor, you could achieve the same thing. Not coincidentally, the capacitors in PSUs are huge, although not battery sized in terms of capacity. The purpose of the capacitors in the PSU is to keep things powered during a power outage to allow for a UPS to switch to battery. The technical term for this is PSU hold up time.

I consider smoothing out high frequency power fluctuations and providing power over a period of minutes that exceeds the capacity of the wall plug to be conceptually two different functions, even if they have similarities.

You’re right that a large enough capacitor could do that, and I’ve worked with high voltage supercapacitor systems which can provide tens of watts for minutes, but the cost is so high that lithium batteries typically make more sense.

So you are saying that Nvidia will finally force USA to the 220V standard? :)

Many American homes already have 240V sockets (eg: NEMA 14-30) for running clothes dryers, car chargers, etc. These can provide over 7200W continuous power!

I guess PC power supplies need to start adopting this standard.

You can't use a NEMA 14-30 to power a PC because 14-30 outlets are split-phase (that's why they have 4 prongs - 2 hot legs, shared neutral, shared ground). To my knowledge, the closest you'll get to split-phase in computing is connecting the redundant PSU in a server to a separate phase or a DC distribution system connected to a multi-phase rectifier, but those are both relegated to the datacenter.

You could get an electrician to install a different outlet like a NEMA 6-20 (I actually know someone who did this) or a European outlet, but it's not as simple as installing more appliance circuits, and you'll be paying extra for power cables either way.

If you have a spare 14-30 and don't want to pay an electrician, you could DIY a single-phase 240v circuit with another center tap transformer, though I wouldn't be brave enough to even attempt this, much less connect a $2k GPU to it.

As far as I’m aware (and as shown by a limited amount of testing that I’ve done myself), any modern PC PSU (with active PFC) is totally fine running on split-phase power: you just use both hots, giving you 240v across them, and the ground. The neutral line is unnecessary.

If you installed a European outlet in a US home then it would be using the same split phase configuration that a NEMA 14-30 does. But many appliances will work just fine, so long as they can handle 60 Hz and don't actually require a distinct neutral and ground for safety reasons. Likewise NEMA 10-30, the predecessor to NEMA 14-30 which is still found in older homes, does not have a ground pin.

I thought the main purpose of providing the neutral line was to be able to power mixed 240V and 120V loads.

I feel like every time I read about USA standards I inevitably discover that any and all SI standards are actually adopted somewhere in the USA - measures in the NASA, 24h clock in the army etc. Just not in the general populace. :)

The entire residential electrical grid in the USA uses 240v, split phase. One hot wire at 120v, one neutral at 0v, and one hot at -120v, out of phase with the other hot. Very rare to have anything else. It’s just that inside the building, the outlets/lights are connected to one side of the split phase connection or the other, giving you only 120v to work with. But then we have special outlets for electric clothes dryers, EV chargers, etc, which give you both hot connections in a single receptacle, for 240v.

PC power supplies already support 240V. Their connectors can take 120V or 240V.

Yes, but a standard household wall socket in the US supplies 120V @ 15A, for a max continuous power of 1.4 kW or so. So typical power supplies are only designed to draw up to that much power, or less.

If someone made a PC power supply designed to plug into a NEMA 14-50 you could run a lot of GPUs! And generate a lot of heat!

You just need the right adapters to connect the C14 connector on most PSUs to NEMA 14-50R. Use these two:

https://www.amazon.com/IronBox-Electric-Connector-Power-Cord...

https://www.amazon.com/14-50P-6-15R-Adapter-Adaptor-Charger/...

As long as the PSU has proper overcurrent protection, you could get away with saying it is designed for this. I suspect you meant designed for higher power draw rather than merely designed to be able to be plugged into the receptacle, but your remark was ambiguous.

Usually, the way people do things to get higher power draw is that they have a power distribution unit that provides C14 receptacles and plugs into a high power outlet like this:

https://www.apc.com/us/en/product/APDU9981EU3/apc-rack-pdu-9...

Then they plug multiple power supplies into it. They are actually able to use the full available AC power this way.

A (small) problem with scaling PSUs to the 50A (40A continuous) that NEMA 14-50 provides is that there is no standard IEC connector for it as far as I know. The common C13/C14 connectors are limited to 10A. The highest is C19/C20 for 16A, which is used by the following:

https://seasonic.com/atx3-prime-px-2200/

https://seasonic.com/prime-tx/

If I read the specification sheets correctly, the first one is exclusively for 200-240VAC while the second one will go to 1600W off 120V, which is permitted by NEMA 5-15 as long as it is not a continuous load.

There is not much demand for higher rated PSUs in the ATX form factor most here would want, but companies without brand names appear to make ones that go up to 3.6kW:

https://www.amazon.com/Supply-Bitcoin-Miners-Mining-180-240V...

As for even higher power ratings, there are companies that make them in non-standard form factors if you must have them. Here is one example:

https://www.infineon.com/cms/en/product/promopages/AI-PSU/#1...

The U.S. has been 240V for over a century. It uses split phase which has opposite phases on each hot line to let you take one and connect it to neutral to get 120V. If you connect both hot lines, you get 240V. For some reason, people in Europe and other places are unaware of this despite this having been the case since electrification happened in the early 20th century.

People are aware of this, but the regular sockets are connected to 120V only. You can (easily) hack an electrical circuit ("consumer unit" in UK) to deliver 240V on an existing plug, but that would be a very serious code violation. SO unless you hack your house circuits, you have 120V on regular sockets.

You can replace the receptacles with ones meant for 240VAC at the same time you modify the wiring. Then it should be okay. Replacing the NEMA 5-15 receptacles with NEMA 6-15 receptacles would work.

Then you could also say that Europe uses 400 V. You get three-phase power with 230 V phases in your home, and high-powered appliances are often designed to use all three phases.

But when people speak of voltages, they usually mean what you get from a typical socket.

This explains why a number of people in Europe seem to love the idea of having triple phase to DC on board chargers on electric cars, even though it makes more sense to have those at the charging points.

That said, a typical socket likely varies more in the U.S. than in Europe since anything that is high draw in the U.S. gets 240VAC while Europe's 220VAC likely suffices for that. I actually have experimented with running some of my computers off 240VAC. It was actually better than 120VAC since the AC to DC conversion is more efficient when stepping down from 240VAC. Sadly, 240VAC UPS units are pricy, so I terminated that in favor of 120VAC until I find a deal on a 240VAC UPS unit.

It's good to know can all heat our bedrooms while mining shitcoins.

Nvidia wants you to buy their datacenter or professional cards for AI. Those often come with better perf/W targets, more VRAM, and better form factors allowing for a higher compute density.

For consumers, they do not care.

PCIe Gen 4 dictates a tighter tolerance on signalling to achieve a faster bus speed, and it took quite a good amount of time for good quality Gen 4 risers to come to market. I have zero doubt in my mind that Gen 5 steps that up even further making the product design just that much harder.

In the server space there is gen 5 cabling but not gen 5 risers.

    > gen 5 cabling

Do you mean OCuLink? Honestly, I never thought about how 1U+ rackmount servers handle PCIe Gen5 wiring/timing issues between NVMe drives (front), GPUs/NICs (rear), and CPUs (middle).

OCuLink has been superseded by MCIO. I was speaking of the custom gen 5 cabled nvme backplane most servers have.

This is the #1 reason why I haven’t upgraded my 2080 Ti. Using my laser printer while my computer is on (even if it’s idle) already makes my UPS freak out.

But NVIDIA is claiming that the 5070 is equivalent to the 4090, so maybe they’re expecting you to wait a generation and get the lower card if you care about TDP? Although I suspect that equivalence only applies to gaming; probably for ML you’d still need the higher-tier card.

The big grain of salt with that "the 5070 performs like a 4090" is that it is talking about having the card fake in 3 extra frames for each one it properly generates. In terms of actual performance boost a 5070 is about 10% faster than a 4070.

Source for your 10% number?

I heard them say that in the Hardware Unboxed youtube video yesterday.

I think it's this one https://youtu.be/olfgrLqtXEo

I don’t see any testing performed in that video. Did I miss it?

No testing, they estimated from the available information.

I think people are speculating based on graphs Nvidia has on their product page.

According to Nvidia [0], DLSS4 with Multi Frame Generation means "15 out of 16 pixels are generated by AI". Even that "original" first out of four frames is rendered in 1080p and AI upscaled. So it's not just 3 extra frames, it's also 75% of the original one.

[0] https://www.nvidia.com/en-us/geforce/news/dlss4-multi-frame-...

Does a laser printer need to be connected to a UPS?

Faulty iron in another room fried my LaserJet. UPS isn't just for loss of power, it should also protect from power spikes. Btw. printer was connected to a (cheap) surge protector strip which didn't help. On positive side nothing else was fried and laser was fixed for 40 euros.

Is it ironic that the electrically expensive part of the LaserJet, the fused, is pretty much an iron in a different format?

https://www.inkguides.com/printer-fuser/

It's not connected to the UPS directly, it's causing voltage dip on the circuit tripping the UPS.

no

I would be careful connecting laser printers to consumer UPS products. On paper all the numbers may line up, but I don't know why you'd want to if you could otherwise avoid it.

If the printer causes your UPS to trip when merely sharing the circuit, imagine the impact to the semiconductors and other active elements when connected as a protected load.

That’s because you have a Brother laser printer which charges its capacitors in the least graceful way possible.

This happens with my Samsung laser printer too, is it not all laser printers?

It's mostly the fuser that is sucking down all the power. In some models, it will flip on and off very quickly to provide a fast warm up (low thermal mass). You can often observe the impact of this in the lights flickering.

Please expand, I am intrigued!

If my Brother laser printer starts while I have the ceiling fan going on the same circuit, the breaker will trip. That's the only thing in my house that will do it. It must be a huge momentary current draw.

Why would you have your laser printer connected to your UPS?

Your UPS is improperly sized. A 5kW Victron Multiplus II with one Pylontech US5000 would cost you around €1600 and should be able to carry all your house, not just your printer.

Thanks for those recommendations. From a few minutes of searching, looks like they would cost 1.5x to 2x that in USA.

We decided to start very small, because we couldn't figure out from the websites of various backup energy installers who was least likely to grossly inflate specs and prices. So, we recently bought a low-end expandable Anker Solix C1000 for around $500 USD as manual reserve power, mainly for the fridge. It seems to be intended more for "glamping", but Anker has good reputation for various unrelated products.

Instead of risers just use pcie ender cords and you can get 4x 3090's working with a creator motherboard (google one that you know can handle 4). You could also use a mining case to do the same.

But, the advantage is that you can load a much more complex model easily (24GB vs 32GB is much easier since 24GB is just barely around 70B parameters).

Yeah, that's bullshit. I have a 3090 and I never want to use it at max power when gaming, because it becomes a loud space heater. I don't know what to do with 575W of heat.

I wonder how many generations it will take until Nvidia launches a graphics card that needs 1kW.

I wish mining was still a thing, it was awesome to have free heating in the cold winter.

Is it not? (Serious question)

Probably not on GPUs - think it all moved to ASICs years ago.

Mining on GPUs was never very profitable unless you held the mined coins for years. I suspect it still is profitable if you are in a position to do that, but the entire endeavor seems extremely risky since the valuation increases are not guaranteed.

Which didn't stop people gobbling up every available gpu in the late 2010's.

(Which, in my opinion, was a contributing factor why VR pc gaming didn't take of when better VR headsets arrived just around that point.)

> Mining on GPUs was never very profitable unless you held the mined coins for years.

If mining is only profitable after holding, it wasn't profitable. Because then you could have spent less money to just buy the coins instead of mining them yourself, and held them afterwards.

you can still fold

Yes but the memory bandwidth of the 5090 is insanely high

soon you'll need to plug your PC into the 240 V dryer outlet lmao

(with the suggested 1000 W PSU for the current gen, it's quite conceivable that at this rate of increase soon we'll run into the maximum of around 1600 W from a typical 110 V outlet on a 15 A circuit)

Yeah. I've been looking at changing out my home lab GPU but I want low power and high ram. NVIDIA hasn't been catering to that at all. The new AMD APUs, if they can get their software stack to work right, would be perfect. 55w TDP and access to nearly 128GB, admittedly at 1/5 the mem bandwidth (which likely means 1/5 the real performance for tasks I am looking at but at 55w and being able to load 128g....)

In theory yes, but it also depends on the workload. RTX 4090 is ranking quite well on the power/performance scale. I'd rather have my card take 400W for 10 minutes to finish the job than take only 200W for 30 minutes.

Sooo much heat .... I'm running a 3080 and playing anything demanding warms my room noticeably.