News for hackers. Stuff that matters.

A comment on MiniGPT-4. Parent comment

Re:MiniGPT-4 by kramerger on Monday April 17, 2023 @07:13AM (#35599171)
Can any of this realistically run on CPU at some point?

(Not training obviously)

[ Parent | on HN ]
Re:MiniGPT-4 by Ambix on Monday April 17, 2023 @07:30AM (#35599241)
I'm developing framework [1] in Golang with this goal in mind :) It successfully runs relatively big LLM right now, and diffusion models will be the next step

[1] https://github.com/gotzmann/llama.go/

[ Parent | on HN ]
Re:MiniGPT-4 by GaggiX on Monday April 17, 2023 @07:29AM (#35599232)
I've run LLaMa models on my CPU before, ViT-L and the Q-former are two transformer models as well, so I can't see why they wouldn't run on a CPU.
[ Parent | on HN ]
Re:MiniGPT-4 by anonzzzies on Monday April 17, 2023 @08:55AM (#35599934)
It will do, probably quite soon. Many people are trying.
[ Parent | on HN ]
Re:MiniGPT-4 by muyuu on Monday April 17, 2023 @09:10AM (#35600101)
quantized Vicuna runs ok-ish in my 16GB i7 laptop (onboard graphics) and the output is usable

see this comparison: https://old.reddit.com/r/LocalLLaMA/comments/12ezcly/compari...

these models quantised to 4bit should run in CPU set ups with 16GB of RAM + 16GB of swap (Linux) and perhaps other setups run similarly

[ Parent | on HN ]
Re:MiniGPT-4 by tantony on Monday April 17, 2023 @09:29AM (#35600385)
Someone is probably going to port it to llama.cpp soon.
[ Parent | on HN ]
Re:MiniGPT-4 by lhl on Monday April 17, 2023 @11:29AM (#35602234)
Yes, you can run inference at decent speeds on CPU with llama.cpp. A token is about 0.75/words, so you can see lots of people getting 4-8 words/s on their CPUs: https://github.com/ggerganov/llama.cpp/issues/34

There a lot of optimizations that can be done. Here's one w/ potentially a 15X AVX speedup for example: https://github.com/ggerganov/llama.cpp/pull/996

[ Parent | on HN ]