Ask HN: Cheapest hardware to run Llama 2 70B

Re:Ask HN: Cheapest hardware to run Llama 2 70B by orost on Wednesday August 09, 2023 @08:12PM (#37070137)

Anything with 64GB of memory will run a quantized 70B model. What else you need depends on what is acceptable speed for you. With a decent CPU but without any GPU assistance, expect output on the order of 1 token per second, and excruciatingly slow prompt ingestion. Any decent Nvidia GPU will dramatically speed up ingestion, but for fast generation, you need 48GB VRAM to fit the entire model. That means 2x RTX 3090 or better. That should generate faster than you can read.

Edit: the above is about PC. Macs are much faster at CPU generation, but not nearly as fast as big GPUs, and their ingestion is still slow.

[ Parent | on HN ]

Re:Ask HN: Cheapest hardware to run Llama 2 70B by tstrimple on Wednesday August 09, 2023 @11:40PM (#37071648)

Do these large models need the equivalent of SLI to take advantage of multiple GPU? Nvidia removed SLI from consumer cards a few years ago so I’m curious whether it’s even an option these days.

[ Parent | on HN ]

Re:Ask HN: Cheapest hardware to run Llama 2 70B by sterlind on Thursday August 10, 2023 @04:46AM (#37073601)

SLI isn't used at all for CUDA. if you meant NVLink, it's apparently not useful at small scales - I think the PCIe lanes are enough.

[ Parent | on HN ]

Re:Ask HN: Cheapest hardware to run Llama 2 70B by ipsum2 on Thursday August 17, 2023 @10:20PM (#37170768)

This is wrong, NVLink is crucial for tensor parallelism in models for training and in large (>40B param) models for inference.

[ Parent | on HN ]

Re:Ask HN: Cheapest hardware to run Llama 2 70B by quickthrower2 on Thursday August 10, 2023 @01:15AM (#37072234)

Vastai would rent you those for about $.50 an hour so gives you an idea of what it costs. Assuming the GPUs memory can be stacked

[ Parent | on HN ]