Re:Lenovo Has a CXL Memory Monster with 128x 128GB DDR5 DIMMs
by HeatrayEnjoyer on Wednesday November 20, 2024 @03:51AM
(#42191960)
A half dozen GPT-4 instances
[ Parent
| on HN ]
Re:Lenovo Has a CXL Memory Monster with 128x 128GB DDR5 DIMMs
by metadat on Wednesday November 20, 2024 @11:39AM
(#42195659)
LLM inference processors (GPUs) don't use DDR, it uses special, costly stacked HBM ram soldered to the board.
I tested out running Llama on a 512GB machine, it's rather slow and inefficient. Maybe 1-token/sec.