News for hackers. Stuff that matters.

A comment on OpenLLM. Parent comment

Re:OpenLLM by moffkalast on Monday June 19, 2023 @05:01AM (#36388848)
Very cool, btw it's not mentioned in the readme so I assume it's only for running full precision models or do quantized GGML/GPTQ/etc. also work with it?
[ Parent | on HN ]
Re:OpenLLM by aarnphm on Monday June 19, 2023 @05:06AM (#36388895)
Hi there, 8bit and 4bit is currently supported on main. GPTQ is working in progress, as well as GGML
[ Parent | on HN ]
Re:OpenLLM by nacs on Monday June 19, 2023 @11:28AM (#36392192)
GPTQ support would be amazing (AutoGPTQ is an easy way to integrate GPTQ support - it's basically just importing autogptq and switching out 1 line in the model loading code).
[ Parent | on HN ]