News for hackers. Stuff that matters.
Re:Llama.rs – Rust port of llama.cpp for fast LLaMA inferenc... by ronsor on Wednesday March 15, 2023 @02:48PM (#35172933)
Oh hey, that's my issue.

It seems like most of the work would simply be moving the inference stuff (feeding tokens to the model and sampling) outside of main(). Most of the other functionality such as model weights loading are already handled in their own functions.

Re:Llama.rs – Rust port of llama.cpp for fast LLaMA inferenc... by tantony on Wednesday March 15, 2023 @04:17PM (#35174153)
I am working on https://github.com/ggerganov/llama.cpp/issues/77 to "librarify" the code a bit. I eventually did want to Rewrite It In Rust™, but OP just beat me to it.