News for hackers. Stuff that matters.

A comment on Code Llama, an AI tool for coding. Parent comment

Re:Code Llama, an AI tool for coding by liuliu on Thursday August 24, 2023 @12:26PM (#37250839)
34B should be able to run on 24GiB consumer graphics card, or 32GiB Mac (M1 / M2 chips) with quantization (5~6bit) (and 7B should be able to run on your smart toaster).
[ Parent | on HN ]
Re:Code Llama, an AI tool for coding by epolanski on Thursday August 24, 2023 @12:57PM (#37251394)
Are there cloud offerings to run those models on somebody's else computer?

Any "eli5" tutorial on how to do so, if so?

I want to give these models a run but I have no powerful GPU to run them on so don't know where to start.

[ Parent | on HN ]
Re:Code Llama, an AI tool for coding by redox99 on Thursday August 24, 2023 @01:07PM (#37251559)
On runpod there is a TheBloke template with everything set up for you. An A6000 is good enough to run 70b 4bit.
[ Parent | on HN ]
Re:Code Llama, an AI tool for coding by kordlessagain on Thursday August 24, 2023 @10:58PM (#37257466)
I started something here about this: https://news.ycombinator.com/item?id=37121384
[ Parent | on HN ]
Re:Code Llama, an AI tool for coding by RossBencina on Friday August 25, 2023 @05:42AM (#37259806)
runpod, togethercomputer, replicate.

Matthew Berman has a tutorial on YT showing how to use TheBloke's docker containers on runpod. Sam Witteveen has done videos on together and replicate, they both offer cloud-hosted LLM inference as a service.

[ Parent | on HN ]