News for hackers. Stuff that matters.
Re:Llama2.c: inference llama 2 in one file of pure C by brucethemoose2 on Sunday July 23, 2023 @08:35PM (#36841580)
Nah, I tried Ubuntu too.

The OpenBLAS package was missing on ARM, along with some other dependencies I needed for compilation.

At the end of the day, even with many tweaks and custom compilation flags, the instance was averaging below 1 token/sec as a Kobold Horde host, which is below the threshold to even be allowed as a llm host.

There are no comments on this comment yet.