News for hackers. Stuff that matters.

A comment on Localllm lets you develop gen AI apps on local CPUs. Parent comment

Re:Localllm lets you develop gen AI apps on local CPUs by brucethemoose2 on Wednesday February 07, 2024 @07:45PM (#39296379)
On the contrary, it's not one-size-fits-all.

It doesn't support everything other backends need (like certain quantization schemes). If it somehow does, then its not universally compatible and that's just going to create more confusion.

[ Parent | on HN ]
Re:Localllm lets you develop gen AI apps on local CPUs by behnamoh on Wednesday February 07, 2024 @11:28PM (#39298143)
It supports grammar and that's more than enough in practice.
[ Parent | on HN ]
Re:Localllm lets you develop gen AI apps on local CPUs by brucethemoose2 on Thursday February 08, 2024 @07:06AM (#39300989)
Grammar has nothing to do with gguf, as it's logit processing (and that is being integrated into other backends as well).
[ Parent | on HN ]