News for hackers. Stuff that matters.

A comment on Open Source AI Is the Path Forward. Parent comment

Re:Open Source AI Is the Path Forward by throwaway_2494 on Tuesday July 23, 2024 @01:03PM (#41048356)
> We’re releasing Llama 3.1 405B

Is it possible to run this with ollama?

[ Parent | on HN ]
Re:Open Source AI Is the Path Forward by vorticalbox on Tuesday July 23, 2024 @01:21PM (#41048555)
If you have the ram for it.

Ollama will offload as many layers as it can to the gpu then the rest will run on the cpu/ram.

[ Parent | on HN ]
Re:Open Source AI Is the Path Forward by jessechin on Tuesday July 23, 2024 @01:20PM (#41048537)
Sure, if you have a H100 cluster. If you quant it to int4 you might get away with using only 4 H100 GPUs!
[ Parent | on HN ]
Re:Open Source AI Is the Path Forward by sheepscreek on Tuesday July 23, 2024 @02:01PM (#41048961)
Assuming $25k a pop, that’s at least $100k in just the GPUs alone. Throw in their linking technology (NVLink) and cost for the remaining parts, won’t be surprised if you’re looking at $150k for such a cluster. Which is not bad to be honest, for something at this scale.

Can anyone share the cost of their pre-built clusters, they’ve recently started selling? (sorry feeling lazy to research atm, I might do that later when I have more time).

[ Parent | on HN ]
Re:Open Source AI Is the Path Forward by tomp on Tuesday July 23, 2024 @03:40PM (#41049959)
you're about right.

https://smicro.eu/nvidia-hgx-h100-640gb-935-24287-0001-000-1

8x H100 HGX cluster for €250k + VAT

[ Parent | on HN ]
Re:Open Source AI Is the Path Forward by rty32 on Tuesday July 23, 2024 @02:20PM (#41049145)
You can rent H100 GPUs.
[ Parent | on HN ]
Re:Open Source AI Is the Path Forward by Havoc on Tuesday July 23, 2024 @06:33PM (#41051686)
If you want your first token around tomorrow lunch sure
[ Parent | on HN ]