News for hackers. Stuff that matters.

A comment on The genie escapes: Stanford copies the ChatGPT AI for less than $600. Parent comment

Re:The genie escapes: Stanford copies the ChatGPT AI for les... by superkuh on Monday March 20, 2023 @05:27PM (#35239016)
As far as I know, yes. https://arxiv.org/abs/2210.17323

"Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline."

This would be 175 billion 3 bit weights instead of 175 billion 16 (or 32!) bit weights. It massively reduces the size of the model. It makes loading it in ram on consumer computers feasible. The number of parameters stays the same.

[ Parent | on HN ]
Re:The genie escapes: Stanford copies the ChatGPT AI for les... by rnosov on Monday March 20, 2023 @05:41PM (#35239175)
> https://arxiv.org/abs/2210.17323

I've read the paper and to be honest I'm not sure what to make of it. Their headline benchmark is perplexity on WikiText2 which would not be particularly relevant to most users. If you look at the tables in appendix A.4 with some more relevant benchmarks you'll sometimes find that straight RTN 4 bit quantisation beats both GPTQ and event full 16 bit original! No explanation of it is given in the paper.

[ Parent | on HN ]
Re:The genie escapes: Stanford copies the ChatGPT AI for les... by sebzim4500 on Monday March 20, 2023 @06:32PM (#35239784)
Some of those benchmarks have a pretty small sample size IIRC, might just be coincidence that the noise introduced by RTN just happens to slightly improve them.

GPTQ beats RTN on almost every benchmark at almost every size, though.

[ Parent | on HN ]
Re:The genie escapes: Stanford copies the ChatGPT AI for les... by rcme on Monday March 20, 2023 @05:49PM (#35239274)
The number of parameters stays the same, but the amount of information encodable by those parameters is not the same.
[ Parent | on HN ]
Re:The genie escapes: Stanford copies the ChatGPT AI for les... by coeneedell on Monday March 20, 2023 @06:49PM (#35239971)
I wonder if reducing the bit depth of parameters like we have been acts as a normalization feature in these huge deep models.
[ Parent | on HN ]
Re:The genie escapes: Stanford copies the ChatGPT AI for les... by thomasahle on Monday March 20, 2023 @07:45PM (#35240527)
But they have to expand it back out to actually use it, right? Or does NVIDIA support 3 bit matrix mult?
[ Parent | on HN ]