Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool
by flangola7 on Tuesday February 14, 2023 @05:15PM
(#34796955)
There's a 20B model now
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool
by smoldesu on Tuesday February 14, 2023 @05:39PM
(#34797253)
The 20B model was the original I believe, and the other models were all pruned from it.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool
by atylerrice on Tuesday February 14, 2023 @10:01PM
(#34799505)
This is incorrect. Thought would be interesting. Pruning usually removes some parameters from the model (setting certain ones to 0 for instance). The smaller models all have less layers and therefore weren’t pruned but were smaller to begin with.
These models are obviously nerfed, but you can run them on budget ARM hardware and get legible, paragraph-length responses in less than 10 seconds.