News for hackers. Stuff that matters.

A comment on Nvidia's Chat with RTX is a promising AI chatbot that runs locally on your PC. Parent comment

Re:Nvidia's Chat with RTX is a promising AI chatbot that run... by FirmwareBurner on Tuesday February 13, 2024 @03:02PM (#39362129)
>LLaMA 13b (24.5GB) and Ministral 7b (13.6GB)

But the HW requirements state 8GB of VRAM. How do those models fit in that?

[ Parent | on HN ]
Re:Nvidia's Chat with RTX is a promising AI chatbot that run... by av3csr on Tuesday February 13, 2024 @06:11PM (#39364155)
They are int4 quantized
[ Parent | on HN ]
Re:Nvidia's Chat with RTX is a promising AI chatbot that run... by Kranar on Tuesday February 13, 2024 @10:54PM (#39366264)
Does int4 mean 4 bits per integer, or 4 bytes/32-bits.

If it means that weights for an LLM can be 4 bits well that's just mind boggling.

[ Parent | on HN ]
Re:Nvidia's Chat with RTX is a promising AI chatbot that run... by sillysaurusx on Tuesday February 13, 2024 @11:40PM (#39366555)
Four bits per parameter. (A parameter is what you call an integer here.)

I was skeptical of it for some time, but it seems to work because individual parameters don’t encode much information. The knowledge is embedded thanks to having a massive number of low bit parameters.

[ Parent | on HN ]
Re:Nvidia's Chat with RTX is a promising AI chatbot that run... by justahuman74 on Tuesday February 13, 2024 @11:42PM (#39366573)
4 bits
[ Parent | on HN ]