News for hackers. Stuff that matters.

A comment on State-of-the-art open-source chatbot, Vicuna-13B, just released model weights. Parent comment

Re:State-of-the-art open-source chatbot, Vicuna-13B, just re... by holoduke on Monday April 03, 2023 @07:56PM (#35433147)
Why would a 4bit quantized model be less accurate than a 16?
[ Parent | on HN ]
Re:State-of-the-art open-source chatbot, Vicuna-13B, just re... by mdaniel on Monday April 03, 2023 @08:53PM (#35433570)
My lay-person's understanding is that it's due to the problem one is trying to solve with a deep learning model: draw a curve through the dimensions which separates "good" from "bad" activation values. The lower resolution the line, the higher likelihood the line will fit sometimes and veer off into erroneous space others

imagine trying to draw the blue line on the right using only lego blocks: https://youtu.be/QDX-1M5Nj7s?t=1202

discussion: https://news.ycombinator.com/item?id=35405338

[ Parent | on HN ]
Re:State-of-the-art open-source chatbot, Vicuna-13B, just re... by dragonwriter on Monday April 03, 2023 @10:02PM (#35434013)
Because 4 bits less precisely specifies the value of the parameter than 16 bits does.
[ Parent | on HN ]