News for hackers. Stuff that matters.
Re:Llama.rs – Rust port of llama.cpp for fast LLaMA inferenc... by mattfrommars on Wednesday March 15, 2023 @10:28PM (#35177435)
Anyone more knowledgeable in this space please explain what is meant by inference?

From what I know. LLaMa is built in Python and assuming PyTorch, does this port in Rust make use of Python process or is it what LLaMA algorithm is fully written in Rust?

Re:Llama.rs – Rust port of llama.cpp for fast LLaMA inferenc... by tantony on Thursday March 16, 2023 @01:40PM (#35185363)
It uses "ggml", a tensor library written in C to do the math. The models weights are converted to a format that can be used by GGML. The higher level structure of the model is created in Rust be leveraging ggml data structures and the weights are passed into the C library.

PyTorch is only needed for reading the original weights while converting them.