News for hackers. Stuff that matters.

A comment on Ask HN: Cheapest hardware to run Llama 2 70B. Parent comment

Re:Ask HN: Cheapest hardware to run Llama 2 70B by tstrimple on Wednesday August 09, 2023 @11:40PM (#37071648)
Do these large models need the equivalent of SLI to take advantage of multiple GPU? Nvidia removed SLI from consumer cards a few years ago so I’m curious whether it’s even an option these days.
[ Parent | on HN ]
Re:Ask HN: Cheapest hardware to run Llama 2 70B by sterlind on Thursday August 10, 2023 @04:46AM (#37073601)
SLI isn't used at all for CUDA. if you meant NVLink, it's apparently not useful at small scales - I think the PCIe lanes are enough.
[ Parent | on HN ]
Re:Ask HN: Cheapest hardware to run Llama 2 70B by ipsum2 on Thursday August 17, 2023 @10:20PM (#37170768)
This is wrong, NVLink is crucial for tensor parallelism in models for training and in large (>40B param) models for inference.
[ Parent | on HN ]