Small offline large language model – TinyChatEngine from MIT

Re:Small offline large language model – TinyChatEngine from MIT by pmontra on Monday December 18, 2023 @02:46AM (#38680117)

I wish that all these repos were more clear about the hardware requirements. Seeing that it runs on a 8 GB Raspberry, probably with abysmal performance, I'd say that it will run on my 32 GB Intel laptop on the CPU. Will it run on its Nvidia card? I remember that the rule of thumb was one GB of GPU RAM per G parameters, so I'd say that it won't run. However this has 4 bit quantization so it could have lower requirements.

Of course the main problem is that I don't know enough about the subject to reason on it on my own.

[ Parent | on HN ]

Re:Small offline large language model – TinyChatEngine from MIT by dkjaudyeqooe on Monday December 18, 2023 @03:20AM (#38680256)

Roughly speaking I believe it's the number of parameters times the size of the parameters. So in the 4 bit case it's half a gigabyte per billion parameters.

From a performance point of view (quantized) integer parameters are going to run better on CPUs than floating point parameters.

[ Parent | on HN ]