Running large language models like ChatGPT on a single GPU

Re:Running large language models like ChatGPT on a single GPU by baobabKoodaa on Tuesday February 21, 2023 @03:19AM (#34878283)

If you are claiming that training a LLM literally only one time is enough and there is no need to train it more than once, you are wrong. The researchers who created OPT didn't go into a basement for 12 months, then come out, train their model once, hit publish, and go to coffee. That is a fantasy. Likewise, if a CS student wants to dabble in this research, they need the ability to train more than once.

I'm not gonna engage in a rhetorical argument about whether this should be called "software development" or "LLM development" or something else. That's unrelated to the question of how much training is required.

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by coldtea on Tuesday February 21, 2023 @04:04AM (#34878630)

>If you are claiming that training a LLM literally only one time is enough and there is no need to train it more than once, you are wrong.

No, I'm rather claiming that what you claimed is wrong in the context of LLM training: "Well maybe not every day, but having a short feedback loop and the ability to run your code multiple times with different variations is generally considered to be a prerequisite for software development".

LLM training is not the same as writing a program and "running your code with different variations". For LLM you don't need to quickly rerun everything with some new corpus - it would be nice, but it's neither a prerequisite, not even crucial for any current use.

Hell, it's not even a "prerequisite" in programming, just good to have. Tons of great programs have been written with very slow build times, without quick edit/compile/build/run cycles.

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by baobabKoodaa on Tuesday February 21, 2023 @05:13AM (#34879170)

I wasn't talking about running the same code with a new corpus. For that kind of use case one can simply fine tune the pretrained model. The example I gave was "if a CS student wants to dabble in this research".

You said "LLM training is not the same as writing a program and running your code with different variations". How do you think these LLMs were made, seriously? Do you think Facebook researchers sat down for 12 months and wrote code non stop without compiling it once, until the program was finished and was used to train the LLM literally only one time?

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by Dylan16807 on Tuesday February 21, 2023 @10:04PM (#34891110)

I would expect them to use small sizes for almost all the testing.

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by baobabKoodaa on Wednesday February 22, 2023 @02:51AM (#34892868)

Yes. There _is_ a need to train LLMs more than once, and training is prohibitively expensive, so you need workarounds such as training on a small subset of data, or a smaller version of the model. We're not yet at the point where a CS student on consumer hardware could afford to do this kind of research.

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by Dylan16807 on Wednesday February 22, 2023 @12:06PM (#34898719)

> We're not yet at the point where a CS student on consumer hardware could afford to do this kind of research.

Okay. But I was saying someone with millions of dollars to spend could do it. And then another poster was arguing that millions of dollars was not enough to be viable because you need lots of repeated runs.

Nobody was saying a student could train one of these models from scratch. The cool potential is for a student to run one, maybe fine tune it.

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by baobabKoodaa on Thursday February 23, 2023 @08:20AM (#34910163)

Here is the upthread comment I was responding to:

> Why would you want to retrain it from scratch every day?

I was explaining why someone might want to retrain it more than once (although not literally every day).

[ Parent | on HN ]