Making AMD GPUs competitive for LLM inference

Re:Making AMD GPUs competitive for LLM inference by superkuh on Wednesday August 09, 2023 @08:47PM (#37070416)

You can say the same thing about a 24GB consumer card. Going from being able to run 13B llamas to 33B doesn't really help you in a commercial sense. This holds true, generally, for other LLM foundational models as well. To do commercial work you're going to need more RAM than consumer cards have. You need at least two if you're going to run the 70B and even then the 70B (and similar) aren't useful commercially. Except in the gathering money from investors who don't know better sense.

[ Parent | on HN ]

Re:Making AMD GPUs competitive for LLM inference by latchkey on Wednesday August 09, 2023 @10:38PM (#37071252)

No one is arguing any of that. You're the one that brought up the 580 specifically.

By the way, still waiting for you to take me up on your 'bet'.

[ Parent | on HN ]

Re:Making AMD GPUs competitive for LLM inference by superkuh on Thursday August 10, 2023 @10:06AM (#37076337)

I was wrong. Sorry. Food trucks do accept cash most places.

Now it's your turn Mr. "You're not going to find rx580's with enough vram for AI. Typically 4-8gb." This is completely false. Rather than acknowledging that you then tried to move the goalposts (much like I did in that past thread saying, "Oh, but maybe it's just my region where they don't.") It looks like we both behave a bit silly when trying to save face when we're wrong.

[ Parent | on HN ]

Re:Making AMD GPUs competitive for LLM inference by latchkey on Thursday August 10, 2023 @10:31AM (#37076631)

> This is completely false.

It isn't completely false. You're doing super limited stuff as a hobbyist that barely works.

[ Parent | on HN ]

Re:Making AMD GPUs competitive for LLM inference by superkuh on Friday August 11, 2023 @12:37AM (#37085161)

The parent article is entirely about running and benchmarking Llama2-7B/13B. This is the "super limited stuff as a hobbyist that barely works" and I've run them at entirely usable speeds on the AMD RX 580. Just admit that you're wrong, or at least that you didn't actually read the article.

[ Parent | on HN ]

Re:Making AMD GPUs competitive for LLM inference by latchkey on Friday August 11, 2023 @01:39PM (#37091487)

"entirely usable" is not the same as "roi efficient"

> from your ignorant perspective

no need for the ad hominem.

[ Parent | on HN ]

Re:Making AMD GPUs competitive for LLM inference by superkuh on Monday August 14, 2023 @02:05PM (#37124369)

Ignorance is not an insult. It just became obvious that you were talking about a different concept (commercial use with big models) than the article itself and everyone else were talking about (7B/13B models). So I generously assumed you just hadn't read it (ignorance). I guess now that you've ignored that and doubled down I can assume you were/are just arguing in bad faith.

[ Parent | on HN ]

Re:Making AMD GPUs competitive for LLM inference by paulmd on Thursday August 10, 2023 @12:00PM (#37077876)

is 70B not commercially useful because of the model storage requirements, or total inference performance, or additional memory per session that's inferencing, or what?

is the output better such that it's desirable, or is this just a case of "too much performance hit for a marginal gain"?

[ Parent | on HN ]