News for hackers. Stuff that matters.

A comment on Show HN: Running LLMs in one line of Python without Docker. Parent comment

Re:Show HN: Running LLMs in one line of Python without Docker by moffkalast on Wednesday October 04, 2023 @05:01PM (#37771727)
Hardly anyone can even run a 70B model, let alone 180B. Any anecdata will be extremely rare.
[ Parent | on HN ]
Re:Show HN: Running LLMs in one line of Python without Docker by jiayq84 on Wednesday October 04, 2023 @05:05PM (#37771791)
In theory one can have 640G = 8 * 80G A100s memory and launch it. 180B Falcon with fp16 will be 360G, so there would be enough memory. It's definitely going to be very expensive indeed.
[ Parent | on HN ]
Re:Show HN: Running LLMs in one line of Python without Docker by sbierwagen on Wednesday October 04, 2023 @08:57PM (#37773697)
Llama.cpp can run quantized Falcon on a top end Mac Studio, which is only five grand: https://twitter.com/ggerganov/status/1699791226780975439

If I'm paying a third party a hundred bucks a month, I'd at least want them to be able to match the capacities of consumer hardware.

[ Parent | on HN ]