News for hackers. Stuff that matters.

A comment on GPU-Accelerated LLM on a $100 Orange Pi. Parent comment

Re:GPU-Accelerated LLM on a $100 Orange Pi by gmiller123456 on Tuesday August 15, 2023 @10:25AM (#37134384)
"Scalability" and "Single Board Computer" don't really belong in the same sentence. That said, today you can get a refurbished mini PC with a lot more power, for a lot less money than the higher end SBCs. But I didn't see any info on how portable this project is to other hardware.
[ Parent | on HN ]
Re:GPU-Accelerated LLM on a $100 Orange Pi by m00x on Tuesday August 15, 2023 @07:39PM (#37140830)
I think the biggest advantage here is that you can run it on the GPU using shared memory, which I'm not sure how widespread it is on mini PCs (at least not intel NUCs).

You could run it using OpenVINO on IntelCPUs, but the performance would probably take a hit. It would be a lot easier though since you can just use ggml.

[ Parent | on HN ]