News for hackers. Stuff that matters.

A comment on Hugging Face Releases Agents. Parent comment

Re:Hugging Face Releases Agents by syntaxing on Wednesday May 10, 2023 @07:36PM (#35895083)
I was so excited until I saw that it's CPU based only. Would you happen to know of any alternative for GPU support, particularly GPTQ models?

Edit: I think textgen itself can support this nowadays

[ Parent | on HN ]
Re:Hugging Face Releases Agents by anton5mith2 on Thursday May 11, 2023 @05:21AM (#35899760)
For now it is CPU only yes, uses AVX instructions. But it's pretty fast anyway, try it out. I have it running on my mbp M1 and it's pretty decent. I think GPU support will come eventually. I wrote an app that uses the openai API and it was nice and simple to just point it at my own local service instead.
[ Parent | on HN ]