Running large language models like ChatGPT on a single GPU

Re:Running large language models like ChatGPT on a single GPU by polishdude20 on Monday February 20, 2023 @01:37PM (#34871178)

There's gotta be something like this already. Like a SETI @ Home type of thing .

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by mryab on Monday February 20, 2023 @01:55PM (#34871420)

There is! See https://petals.ml/ for inference of models like BLOOM-176B over the internet or https://arxiv.org/abs/2301.11913 and https://arxiv.org/abs/2206.01288 that show you how to do pretraining from scratch in the same setting. Disclaimer: I'm a coauthor of these systems (including the one in OP)

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by exebook on Monday February 20, 2023 @03:26PM (#34872464)

Amazing work! If I had a GPU, I'd join. I know similar project for text-to-image: https://aqualxx.github.io/stable-ui/

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by efreak on Tuesday February 21, 2023 @09:59PM (#34891080)

https://github.com/aqualxx/stable-horde-notebook

My only problem with stable horde is that their anti-cp measure involves checking the prompt for words like small, meaning I can't use a nsfw-capable model with certain prompts (holding a very small bag, etc). That, and seeing great things in the image rating and being unable to reproduce because it doesn't provide the prompt.

[ Parent | on HN ]