Re:Nvidia announces next-gen RTX 5090 and RTX 5080 GPUs
by terhechte on Tuesday January 07, 2025 @10:06AM
(#42623111)
If you have 128gb ram, try running MoE models, they're a far better fit for Apple's hardware because they trade memory for inference performance. using something like Wizard2 8x22b requires a huge amount of memory to host the 176b model, but only one 22b slice has to be active at a time so you get the token speed of a 22b model.
[ Parent
| on HN ]
Re:Nvidia announces next-gen RTX 5090 and RTX 5080 GPUs
by FuriouslyAdrift on Tuesday January 07, 2025 @10:30AM
(#42623385)
Project Digits... https://www.nvidia.com/en-us/project-digits/
Re:Nvidia announces next-gen RTX 5090 and RTX 5080 GPUs
by throwaway48476 on Tuesday January 07, 2025 @04:10PM
(#42627511)
I guess they're tired of people buying macs for AI.
Re:Nvidia announces next-gen RTX 5090 and RTX 5080 GPUs
by logankeenan on Tuesday January 07, 2025 @11:30AM
(#42624227)
Do you have any recommendations on models to try?
Re:Nvidia announces next-gen RTX 5090 and RTX 5080 GPUs
by stkdump on Tuesday January 07, 2025 @12:03PM
(#42624657)
Mixtral and Deepseek use MOE. Most others don't.
Re:Nvidia announces next-gen RTX 5090 and RTX 5080 GPUs
by Terretta on Tuesday January 07, 2025 @12:51PM
(#42625189)
Mixtral 8x22b https://mistral.ai/news/mixtral-8x22b/
Re:Nvidia announces next-gen RTX 5090 and RTX 5080 GPUs
by terhechte on Tuesday January 07, 2025 @01:10PM
(#42625397)
In addition to the ones listed by others, WizardLM2 8x22b (was never officially released by Microsoft but is available).
Re:Nvidia announces next-gen RTX 5090 and RTX 5080 GPUs
by memhole on Tuesday January 07, 2025 @02:38PM
(#42626441)
I planted garlic this year. Thanks for documenting! I can’t wait to see what I get harvest time.
I like the Llama models personally. Meta aside. Qwen is fairly popular too. There’s a number of flavors you can try out. Ollama is a good starting point to try things quickly. You’re def going to have to tolerate things crashing or not working imo before you understand what your hardware can handle.
Re:Nvidia announces next-gen RTX 5090 and RTX 5080 GPUs
by cma on Tuesday January 07, 2025 @10:59AM
(#42623787)
You can also run the experts on separate machines with low bandwidth networking or even the internet (token rate limited by RTT)
Re:Nvidia announces next-gen RTX 5090 and RTX 5080 GPUs
by memhole on Tuesday January 07, 2025 @02:32PM
(#42626352)
I haven’t had great luck with the wizard as a counter point. The token generation is unbearably slow. I might have been using too large of a context window, though. It’s an interesting model for sure. I remember the output being decent. I think it’s already surpassed by other models like Qwen.
Re:Nvidia announces next-gen RTX 5090 and RTX 5080 GPUs
by terhechte on Tuesday January 07, 2025 @03:21PM
(#42626975)
Long context windows are a problem. I gave Qwen 2.5 70b a ~115k context and it took ~20min for the answer to finish.
The upside of MoE models vs 70b+ models is that they have much more world knowledge.