Re:Cerebras-GPT vs. LLaMA AI Model Performance Comparison
by sbierwagen on Friday March 31, 2023 @01:38AM
(#35383053)
I'm inclined to think that the model is just too small, or that integer quantization damaged it badly. It has the GPT-2 trait of getting "stuck", (mode collapse?) if it gives a bad response once, it will tend to repeat itself over and over. (Repetition was also a bad GPT-2 trait)
It's important to note that 7 billion parameters really is very small. 20 times smaller than GPT-3. Smaller still than ChatGPT or GPT-4. I find it plausible that in the future there will be distilled models substantially smaller than GPT-3 but with all its power, but GPT4ALL-LLaMA-7B isn't it.
It's important to note that 7 billion parameters really is very small. 20 times smaller than GPT-3. Smaller still than ChatGPT or GPT-4. I find it plausible that in the future there will be distilled models substantially smaller than GPT-3 but with all its power, but GPT4ALL-LLaMA-7B isn't it.