If you have 128gb ram, try running MoE models, they're a far better fit for Apple's hardware because they trade memory for inference performance. using something like Wizard2 8x22b requires a huge amount of memory to host the 176b model, but only one 22b slice has to be active at a time so you get the token speed of a 22b model.

Project Digits... https://www.nvidia.com/en-us/project-digits/

I guess they're tired of people buying macs for AI.

Do you have any recommendations on models to try?

Mixtral and Deepseek use MOE. Most others don't.

Mixtral 8x22b https://mistral.ai/news/mixtral-8x22b/

In addition to the ones listed by others, WizardLM2 8x22b (was never officially released by Microsoft but is available).

I planted garlic this year. Thanks for documenting! I can’t wait to see what I get harvest time.

I like the Llama models personally. Meta aside. Qwen is fairly popular too. There’s a number of flavors you can try out. Ollama is a good starting point to try things quickly. You’re def going to have to tolerate things crashing or not working imo before you understand what your hardware can handle.

You can also run the experts on separate machines with low bandwidth networking or even the internet (token rate limited by RTT)

I haven’t had great luck with the wizard as a counter point. The token generation is unbearably slow. I might have been using too large of a context window, though. It’s an interesting model for sure. I remember the output being decent. I think it’s already surpassed by other models like Qwen.

Long context windows are a problem. I gave Qwen 2.5 70b a ~115k context and it took ~20min for the answer to finish. The upside of MoE models vs 70b+ models is that they have much more world knowledge.

They are intentionally keeping the VRAM small on these cards to force people to buy their larger, more expensive offerings.

No gamers need such high VRAM, if you're buying Gaming cards for ML work you're doing it wrong.

It's Nvidia that considers them, "gaming cards". The market decides their use in reality though.

Their strategy is to sell lower-VRAM cards to consumers with the understanding that they can make more money on their more expensive cards for professionals/business. By doing this, though they're creating a gap in the market that their competitors could fill (in theory).

Of course, this assumes their competitors have half a brain cell (I'm looking at YOU, Intel! For fuck's sake give us a 64GB ARC card already!).

And if you buy the cards that Nvidia says are for gaming and then complain that they don't have good specs for ML, who is the fool exactly?

Games already exceed 16 GBs at 4k from years.

I exceed 16GB in Chrome.

That says more about Chrome than anything else.

I use Firefox and have an 8Gb card and only encounter problems when I have more than about 125 windows with about 10-20 tabs each.

Yes, I am a tab hoarder.

And yes, I am going to buy a 16Gb card soon. :P

1200 tabs isn't that many

That's why I need to get a 16Gb card. :P

System Ram != GPU VRAM

> Gaming cards for ML work you're doing it wrong

lol okay. "doing it wrong" for a tenth of the cost.

And screwing gamers over by raising the prices by 2x. Fuck that.

Believe it or not, it's possible to be interested in both machine learning and videogames. That's ignoring the notion that it's somehow how screwing over gamers. Buy a fucking AMD card. They're great at gaming and you don't need CUDA anyways. Enjoy the long-term acceleration of GPU performance increases you're getting by the way. All that stuff comes from innovations made for workstation/DL setups.

Get an AMD GPU? Said no one ever.

It seems like the 90-series cards are going to be targeting prosumers again. People who play games but may use their desktop for work as well. Some people are doing AI training on some multiple of 3090/4090 today but historically the Titan cards that preceded the 90s cards were used by game developers, video editors and other content developers. I think NVIDIA is going to try to move the AI folks onto Digits and return the 90-series back to its roots but also add in some GenAI workloads.

forget the post but some dude had a startup piping his 3090 to use via cloudflare tunnels for his ai saas making 5 figures a month off of his 1k gpu that handled the work load, I'd say he was doing it more then right.

And if his volume grows 100x should we expect him to run his company off gaming gpus? Just because you can do something doesn't mean you should or that it's ideal.

There's a reason large companies are buying H100s and not 4090s. Despite what you guys think, serious ML work isn't done on the consumer cards for many reasons: FP16/FP8 TFLOPS, NVLINK, power consumption, physical space, etc.

MS Flight Simulator 2024 can consume...who knows how much.

I know my 10 GB 3080 ran out of VRAM playing it on Ultra, and i was getting as low as 2 fps because I'm bottlenecked by the PCI-Express bus as it has to constantly page the entire working set of textures and models in and out.

I'm getting a 5090 for that, plus I want to play around with 7B parameter LLMs and don't want to quantize below 8 bits if I can help it.

I've regularly exceeded 24 GiB of VRAM in Microsoft Flight Simulator 2024. Imagine a huge airport environment with high levels of detail, plus AI aircraft in the ground and sky. Then, on top of that, terrain and textures of the surrounding environment.

And that's at 1440p, not even 4K. The resulting stutters are... not pretty.

tell us how to do it right.

Get your daddy's credit card and buy H100s like a serious person.

If the VRAM wasn't small, the cards would all get routed to non gaming uses. Remember the state of the market when the 3000 series was new?

Then they should sell more of them.

They can only make so many, that's part of the problem

They should contact Intel.

Why sell more when you can sell less for more

Maybe, but if they strapped these with 64gb+ wouldn’t that be wasted on folks buying it for its intended purpose? Gaming. Though the “intended use” is changing and has been for a bit now.

The only reason gaming doesn't use all the VRAM is because typically GPUs don't have all the VRAM. If they did then games would somehow find a way to use it.

Game engines are optimized for lowest common denominator, being in this case consoles. PC games are rarely exclusivities, so same engine has to make it running with least ram available and differences between versions are normally small.

One normally uses some ultra texture pack to utilize current gen card's memory fully on many games.

Consoles would have more VRAM too if these cards had more VRAM. It's not like they're made separately in isolation.

XX90 is only half a gaming card it's also the one the entire creative professional 3D CGI, AI, game dev industry runs on.

Not really, the more textures you can put into memory the faster they can do their thing.

PC gamers would say that a modern mid-range card (1440p card) should really have 16GB of vram. So a 5060 or even a 5070 with less than that amount is kind of silly.

hmmm, maybe they can had different offerings like 16GB, 32GB, 64GB, etc. Maybe we can even have 4 wheels on a car.

So you're saying more VRAM costs more money? What a novel idea!

Conversely, this means you can pay less if you need less.

Seems like a win all around.

Saw someone else point out that potentially the culprit here isn’t nvidia but memory makers. It’s still 2gb per chip and has been since forever

GDDR7 apparently has the capability of 3gb per chip. As it becomes more available their could be more VRAM configurations. Some speculate maybe an RTX 5080 Super 24gb release next year. Wishful thinking perhaps.

Totally agree. I call this the "Apple Model". Just like the Apple Mac base configurations with skimpy RAM and Drive capacities to make the price look "reasonable". However, just like Apple, NVIDIA does make really good hardware.

Makes sense. The games industry doesn't want another crypto mining-style GPU shortage.

Is there actually less VRAM on the cards or is it just disabled?

GPU manufacturers have no reason to include additional memory chips of no use on a card.

This isn't like a cutdown die, which is a single piece with disabled functionality...the memory chips are all independent (expensive) pieces soldered on board (the black squares surrounding the GPU core):

https://cdn.mos.cms.futurecdn.net/vLHed8sBw8dX2BKs5QsdJ5-120...

Well, they are gaming cards. 32GB is plenty for that.

If you want to run LLMs buy their H100/GB100/etc grade cards. There should be no expectation that consumer grade gaming cards will be optimal for ML use.

Yes there should be. We don’t want to pay literal 10x markup because the card is suddenly “enterprise”.

Totally unreasonable expectation. Sry. The cards are literally built for gamers for gaming. That they work for ML is a happy coincidence.

You can’t possibly be naive enough to believe that Nvidia’s Titan class cards were designed exclusively for gamers.

> There should be no expectation that consumer grade gaming cards will be optimal for ML use.

And yet it just so happens they work effectively the same. I've done research on an RTX 2070 with just 8 GB VRAM. That card consistently met or got close to the performance of a V100 albeit with less vram.

Why indicate people shouldn't use consumer cards? It's dramatically (like 10x-50x) cheaper. Is machine learning only for those who can afford 10k-50k USD workstation GPU's? That's lame and frankly comes across as gate keeping.

Honestly I can't really imagine how a person could reasonably have this stance. Just let folks buy hardware and use it however they want. Sure if may be less than optimal but it's important to remember that not everyone in the world has the money to afford an H100.

Perhaps you can explain some other better reason for why people shouldn't use consumer cards for ML? It's frankly kind of a rude suggestion in the absence of a better explanation.

If you can do research on a mid tier consumer card then more power to you. I'm specifically referencing the people who are complaining that the specs on consumer video game GPUs are not good for ML work. Like theres just no reasonable expectation that they will be.

Ah, I see what you mean. Yeah I think it comes from a place of viewing increase in VRAM as relatively low cost and therefore an artificial limitation of sorts used to differentiate between consumer and workstation products (and the respective price disparities).

Which may be true although there are more differences than just VRAM and I assume those market segments have different perceptions of the real value Gamers want it cheaper/faster, institutions want it closer to state of the art, more robust to lengthy workloads (as in year long training sessions), and better support from nvidia. Among other things.

Check out their project digits announcement, 128GB unified memory with infiniband capabilities for $3k.

For more of the fast VRAM you would be in Quadro territory.

> after running LLMs with 128 GB RAM on the M3 Max,

These are monumentally different. You cannot use your computer as an LLM. Its more novelty.

I'm not even sure why people mention these things. Its possible, but no one actually does this out of testing purposes.

It falsely equates Nivida GPUs with Apple CPUs. The winner is Apple.

Why are transformers a better fit for frame generation. Is it because they can better utilize context from the previous history of frames ?