Re:Llama2.c: inference llama 2 in one file of pure C
by jvickers on Sunday July 23, 2023 @07:48PM
(#36841277)
Is it any easier when using Ubuntu on ARM Oracle servers?
Re:Llama2.c: inference llama 2 in one file of pure C
by brucethemoose2 on Sunday July 23, 2023 @08:35PM
(#36841580)
Nah, I tried Ubuntu too.
The OpenBLAS package was missing on ARM, along with some other dependencies I needed for compilation.
At the end of the day, even with many tweaks and custom compilation flags, the instance was averaging below 1 token/sec as a Kobold Horde host, which is below the threshold to even be allowed as a llm host.
Re:Llama2.c: inference llama 2 in one file of pure C
by pedrovhb on Sunday July 23, 2023 @09:04PM
(#36841784)
Not sure if that's still the case. I remember having trouble building it a couple of months ago, had to tweak the Makefile because iirc it assumed ARM64 <=> Mac, but I recently re-cloned the repo and started from scratch and it was as simple as `make DLLAMA_BLAS=1`. I don't think I have any special setup other than having installed the apt openblas dev package.
Re:Llama2.c: inference llama 2 in one file of pure C
by brucethemoose2 on Sunday July 23, 2023 @10:26PM
(#36842413)
IDK. A bunch of basic development packages like git were missing from my Ubuntu image when I tried last week, and I just gave up because it seemed like a big rabbit hole to go down.
I can see the ARM64 versions on the Ubuntu web package list, so... IDK what was going on?
On Oracle Linux, until I changed some env variables and lines in the makefile, the openblas build would "work," but it was actually silently failing and not using OpenBLAS.
Also its really tricky to even build llama.cpp with a BLAS library, to make prompt ingestion less slow. The Oracle Linux OpenBLAS build isnt detected ootb, and it doesn't perform well compared to x86 for some reason.
LLVM/GCC have some kind of issue identifying the Ampere ARM architecture (march=native doesn't really work), so maybe this could be improved with the right compiler flags?