News for hackers. Stuff that matters.

A comment on OpenLLM. Parent comment

Re:OpenLLM by philoko on Monday June 19, 2023 @09:25AM (#36390819)
For Falcon 40b, the 8-bit version would probably need about 48GB of VRAM while the 4-bit would need something closer to 28GB.
[ Parent | on HN ]
Re:OpenLLM by aarnphm on Monday June 19, 2023 @12:47PM (#36393230)
Currently on main, 8bit and 4bit quant is supported

One can simply do

```openllm start falcon --model-id tiiuae/falcon-40b-instruct --quantize int4```

Beware that there is no free lunch, meaning the quality of inference will degrade by alot when using int 4 quantization

[ Parent | on HN ]