How Is LLaMa.cpp Possible?

Re:How Is LLaMa.cpp Possible? by redox99 on Tuesday August 15, 2023 @07:58PM (#37140967)

>Still, we are looking at GPT3 level of performance on one server rack. That says something when less than a year ago, such AI was literally magic and only run on a massive datacenter.

I'm not sure what you mean by this. You've always been able to run GPT3 on a single server (your typical 8xA100).

[ Parent | on HN ]

Re:How Is LLaMa.cpp Possible? by blovescoffee on Tuesday August 15, 2023 @08:10PM (#37141056)

Am I missing something or how do you know this? Also I think the OP was talking about a single card not multiple but that was just my reading.

[ Parent | on HN ]

Re:How Is LLaMa.cpp Possible? by redox99 on Tuesday August 15, 2023 @08:15PM (#37141096)

Because 175B parameters (350GB for the weights FP16, let's say a bit over 400GB for actual inference), fit very comfortably on 8xA100 (640GB VRAM total).

And basically all servers will have 8xA100 (maybe 4xA100). Nobody bothers with a single A100 (of course in a VM you might have access to only one)

[ Parent | on HN ]

Re:How Is LLaMa.cpp Possible? by axiom92 on Tuesday August 15, 2023 @08:47PM (#37141353)

> And basically all servers will have 8xA100

for those wondering: no this is not the norm. My lab at CMU doesn't own any A100s (we have A6000s).

[ Parent | on HN ]

Re:How Is LLaMa.cpp Possible? by doctorpangloss on Tuesday August 15, 2023 @08:57PM (#37141430)

The servers the commenter is talking about are DGX machines from NVIDIA.

It doesn’t really make sense to BTO. What you gain economically you lose in the science you can do.

But nobody could have anticipated this.

[ Parent | on HN ]

Re:How Is LLaMa.cpp Possible? by _zoltan_ on Wednesday August 16, 2023 @04:57AM (#37144436)

you could also get HGX from any of the vendors.

[ Parent | on HN ]

Re:How Is LLaMa.cpp Possible? by bobboies on Wednesday August 16, 2023 @07:43AM (#37145580)

Wtf does HGX mean? God enough with the acronyms people.

Please take an extra ten seconds to speak in proper human language!

You could save on the worlds carbon footprint by reducing the number of times humans have to search for “what is NVIDIA hgx?” or is it “what is AMD HGX” and then subsequently visiting the websites to see if that’s right or not.

[ Parent | on HN ]

Re:How Is LLaMa.cpp Possible? by Vvector on Wednesday August 16, 2023 @10:08AM (#37147248)

What does Wft mean? God enough with the acronyms people. /s

[ Parent | on HN ]

Re:How Is LLaMa.cpp Possible? by bobboies on Wednesday August 16, 2023 @04:14PM (#37153077)

You got me there hahaha

However, there’s a difference between an acronym known to the broader public versus some single shot, context-specific one!

[ Parent | on HN ]

Re:How Is LLaMa.cpp Possible? by _zoltan_ on Wednesday August 16, 2023 @04:58AM (#37144443)

who's norm? I assure you it's the norm. :)

[ Parent | on HN ]

Re:How Is LLaMa.cpp Possible? by loxias on Thursday August 17, 2023 @03:55AM (#37158532)

> And basically all servers will have 8xA100 (maybe 4xA100). Nobody bothers with a single A100 (of course in a VM you might have access to only one)

wishing, or guessing something, without actual experience, doesn't make it true.

[ Parent | on HN ]

Re:How Is LLaMa.cpp Possible? by brucethemoose2 on Tuesday August 15, 2023 @10:54PM (#37142341)

8xA100 is technically a single server, but I think OP is talking about affordable and plentiful CPU hosts, or even relatively modest single GPU instances.

DGX boxes do not grow on trees, especially these days

[ Parent | on HN ]