Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on OpenLLM. Parent comment
One can simply do
```openllm start falcon --model-id tiiuae/falcon-40b-instruct --quantize int4```
Beware that there is no free lunch, meaning the quality of inference will degrade by alot when using int 4 quantization
There are no comments on this comment yet.
One can simply do
```openllm start falcon --model-id tiiuae/falcon-40b-instruct --quantize int4```
Beware that there is no free lunch, meaning the quality of inference will degrade by alot when using int 4 quantization