News for hackers. Stuff that matters.

A comment on Alpaca- An Instruct Tuned Llama 7B. Responses on par with txt-DaVinci-3. Demo up. Parent comment

Re:Alpaca- An Instruct Tuned Llama 7B. Responses on par with... by magicalhippo on Monday March 13, 2023 @07:12PM (#35144546)
Some parameters would be more sensitive than others I suppose? So could you use 4 bits for most, and 8 bits, or even 16, for the remaining?
[ Parent | on HN ]
Re:Alpaca- An Instruct Tuned Llama 7B. Responses on par with... by ch33zer on Monday March 13, 2023 @07:36PM (#35144860)
I know nothing about this so my opinion means little, but I imagine it's hard to know which parameters are important enough to use more bits for.

I do wonder if it would be possible to have the model determine during training how important each parameter is, while maybe rewarding it for having more small parameters?

[ Parent | on HN ]
Re:Alpaca- An Instruct Tuned Llama 7B. Responses on par with... by dongping on Monday March 13, 2023 @08:01PM (#35145171)
That's exactly why bitsandbytes has a threshold parameter to control the quantization.
[ Parent | on HN ]
Re:Alpaca- An Instruct Tuned Llama 7B. Responses on par with... by magicalhippo on Monday March 13, 2023 @08:11PM (#35145269)
Nice, good to know, thanks!
[ Parent | on HN ]