Interesting, what RAM do you use exactly? 24x 16GB DDR5-6000 DIMMs? It seems that those boards only support up to DDR5-4800: https://geizhals.de/?cat=mbsp3&xf=4921_2%7E493_24x+DDR5+DIMM...
Does the core count matter or can you get away with the smallest 2x EPYC 9015 configuration? What are "good speeds"?
Looks like we will soon get boards supporting 24x DDR5-6000 for the EPYC 9005 CPUs.
I use 24 sticks of ddr5-4800, which gets me up to 9t/s on deepseek 2.5 at q8. 48 threads was optimal in llama.cpp. I would like to move to epyc 9005 chips and ddr5-6000, but it is cost prohibitive with CPUs still over $10k each on eBay.
I followed the guide at https://rentry.co/miqumaxx/
How many cores do your CPUs have? Are you using the 64 core EPYC 9334 mentioned in the linked page? Do that many cores provide a speedup versus having fewer cores?
Yes I have the same engineering sample. I can only keep 48 cores fed with max memory bandwidth given current llama.cpp architectural constraints
Re:Tencent Hunyuan-Large
by ConnectX-2 on Sunday November 17, 2024 @04:11AM
(#42162968)
>I only use the GPU for context processing.
If your GPU has enough VRAM to support it, you might benefit from https://github.com/kvcache-ai/ktransformers
They're actually a best-case for CPU inference vs dense models. I usually run deepseek 2.5 quanted to q8, but if this model works well I'll probably switch to it once support hits llama.cpp.