News for hackers. Stuff that matters.
Re:Lenovo Has a CXL Memory Monster with 128x 128GB DDR5 DIMMs by HeatrayEnjoyer on Wednesday November 20, 2024 @03:51AM (#42191960)
A half dozen GPT-4 instances
Re:Lenovo Has a CXL Memory Monster with 128x 128GB DDR5 DIMMs by metadat on Wednesday November 20, 2024 @11:39AM (#42195659)
LLM inference processors (GPUs) don't use DDR, it uses special, costly stacked HBM ram soldered to the board.

I tested out running Llama on a 512GB machine, it's rather slow and inefficient. Maybe 1-token/sec.