News for hackers. Stuff that matters.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by anentropic on Tuesday February 14, 2023 @11:17AM (#34791550)
There's a float16 version of GPT-J rewritten in C++ here that apparently fits into 16GB...

https://github.com/ggerganov/ggml/tree/master/examples/gpt-j

it's CPU-only and runs inference at "about ~6 words per second" on an M1 MBP

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by smoldesu on Tuesday February 14, 2023 @12:02PM (#34792322)
That's for the max-size model (6B parameters). GPT-Neo has pruned models 125m and 1.3b that fit into 1.5gb and 6gb of memory respectively.

These models are obviously nerfed, but you can run them on budget ARM hardware and get legible, paragraph-length responses in less than 10 seconds.

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by flangola7 on Tuesday February 14, 2023 @05:15PM (#34796955)
There's a 20B model now
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by smoldesu on Tuesday February 14, 2023 @05:39PM (#34797253)
The 20B model was the original I believe, and the other models were all pruned from it.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by atylerrice on Tuesday February 14, 2023 @10:01PM (#34799505)
This is incorrect. Thought would be interesting. Pruning usually removes some parameters from the model (setting certain ones to 0 for instance). The smaller models all have less layers and therefore weren’t pruned but were smaller to begin with.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by mark_l_watson on Tuesday February 14, 2023 @02:41PM (#34794813)
They have a link to the bottom for something similar, but with GPT-2. The Hugging Face docs have hints for using FP16, and avoiding double memory requirements for model and checkpoint data.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by atylerrice on Tuesday February 14, 2023 @10:03PM (#34799522)
You can also use int8 quantization which comes for free with huggingface and isn’t a huge cost to accuracy. Which cuts the memory usage almost in half again. (almost in half because some layers might stay 32 bit if they make a tiny part of the model)