News for hackers. Stuff that matters.

A comment on Localllm lets you develop gen AI apps on local CPUs.

Re:Localllm lets you develop gen AI apps on local CPUs by andy99 on Wednesday February 07, 2024 @06:30PM (#39295751)
One thing I'm happy about is that the gguf file format is getting widely used, here endorsed by Google. I vastly prefer working with gguf and have adopted it for my project, hopefully it gets the critical mass to become a "native" format and we can stop with pytorch checkpoints.
[ Parent | on HN ]
Re:Localllm lets you develop gen AI apps on local CPUs by brucethemoose2 on Wednesday February 07, 2024 @07:45PM (#39296379)
On the contrary, it's not one-size-fits-all.

It doesn't support everything other backends need (like certain quantization schemes). If it somehow does, then its not universally compatible and that's just going to create more confusion.

[ Parent | on HN ]
Re:Localllm lets you develop gen AI apps on local CPUs by behnamoh on Wednesday February 07, 2024 @11:28PM (#39298143)
It supports grammar and that's more than enough in practice.
[ Parent | on HN ]
Re:Localllm lets you develop gen AI apps on local CPUs by brucethemoose2 on Thursday February 08, 2024 @07:06AM (#39300989)
Grammar has nothing to do with gguf, as it's logit processing (and that is being integrated into other backends as well).
[ Parent | on HN ]