News for hackers. Stuff that matters.
Re:Open source solution replicates ChatGPT training process by Taek on Monday February 20, 2023 @10:57AM (#34869302)
For reference, GPT-NeoX is a 20B parameter model, and it runs on 45 GB of VRAM. On an 80 GB A100 you could probably run a 35B parameter model. Maybe 8 A100 cards to do inference on ChatGPT?

Or 32 3090 cards, which would run you under $40k total.

Re:Open source solution replicates ChatGPT training process by fswd on Tuesday February 21, 2023 @01:40PM (#34884961)
20B GPT-NeoX runs on a 3090 in 8 bit mode