News for hackers. Stuff that matters.

A comment on Running large language models like ChatGPT on a single GPU.

Re:Running large language models like ChatGPT on a single GPU by lxe on Monday February 20, 2023 @03:15PM (#34872334)
Got the ops-6.7b chatbot running on a windows machine with a 3090 in mere minutes. The only difference was to install the cuda pytorch `pip install torch==1.13.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117` just like in stable diffusion's case.

It performs as expected:

    Human: Tell me a joke
    Machine: I have no sense of humour

    Human: What's 2+5?
    Machine: I cannot answer that.
[ Parent | on HN ]
Re:Running large language models like ChatGPT on a single GPU by rjb7731 on Monday February 20, 2023 @03:41PM (#34872637)
Looks like its no bueno on google colab, also it takes prompts in via input() too rather then a command line argument.
[ Parent | on HN ]
Re:Running large language models like ChatGPT on a single GPU by hackernewds on Tuesday February 21, 2023 @12:40PM (#34884090)
Could it work on Google Colab?
[ Parent | on HN ]
Re:Running large language models like ChatGPT on a single GPU by Ajedi32 on Monday February 20, 2023 @03:33PM (#34872548)
6.7b is pretty small, no? Do you even need offloading for that on a 3090? I'd be curious to see what's needed to run opt-30b or opt-66b with reasonable performance. The README suggests that even opt-175b should be doable with okay performance on a single NVIDIA T4 if you have enough RAM.
[ Parent | on HN ]
Re:Running large language models like ChatGPT on a single GPU by nathan_compton on Monday February 20, 2023 @04:07PM (#34872915)
It is entirely possible to run 6.7B parameter models on a 3090, although I believe you need 16 bit weights. I think you can squeeze a 20b parameter model onto the 3090 if you go all the way down to 8.
[ Parent | on HN ]
Re:Running large language models like ChatGPT on a single GPU by A4ET8a8uTh0 on Monday February 20, 2023 @03:30PM (#34872523)
Hey. So did anyone try doing it with AMD cards ( I know Nvidia seems preferable now )?
[ Parent | on HN ]