News for hackers. Stuff that matters.
Re:Llama.rs – Rust port of llama.cpp for fast LLaMA inferenc... by tantony on Thursday March 16, 2023 @01:40PM (#35185363)
It uses "ggml", a tensor library written in C to do the math. The models weights are converted to a format that can be used by GGML. The higher level structure of the model is created in Rust be leveraging ggml data structures and the weights are passed into the C library.

PyTorch is only needed for reading the original weights while converting them.

There are no comments on this comment yet.