Running large language models like ChatGPT on a single GPU

Re:Running large language models like ChatGPT on a single GPU by SomeHacker44 on Monday February 20, 2023 @03:12PM (#34872308)

I do not know anyone offhand with a 200+GB RAM home computer. The GPU is not all that is needed; you need to keep the parameters and other stuff in memory too.

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by Filligree on Monday February 20, 2023 @03:21PM (#34872410)

Running it off a fast NVMe apparently works. I don't know what the performance is like, though.

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by zargon on Monday February 20, 2023 @04:02PM (#34872868)

256gb of ddr4 rdimms only costs about $400 right now. $200 for ddr3. Not uncommon in homelabs.

I don't think 200gb ram is actually required, that's just what that cloud vm was spec'd with. Though the 175b model should see benefit with ram even beyond 200gb.

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by woadwarrior01 on Monday February 20, 2023 @05:59PM (#34874062)

I own a two year old headless home computer with 256GB of RAM and two 3090s. I ssh into it from my mac to run ML training jobs.

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by hedgehog0 on Tuesday February 21, 2023 @06:18AM (#34879683)

What’s the price may I ask

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by alex_sf on Tuesday February 21, 2023 @12:11PM (#34883656)

I have a similar setup. GPUs were $1.6k US, rest of the system was another ~$1k. This includes 256GB RAM and dual Xeons.

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by zac_hudson on Tuesday February 21, 2023 @04:41PM (#34887863)

I bought an old server from ebay, and ripped the guts out to make a desktop computer. 128GB of ddr3 ram and 48 E5 v2 xeons for 300$.

[ Parent | on HN ]