Re:Llama.rs – Rust port of llama.cpp for fast LLaMA inferenc...
by tantony on Thursday March 16, 2023 @01:40PM
(#35185363)
It uses "ggml", a tensor library written in C to do the math. The models weights are converted to a format that can be used by GGML. The higher level structure of the model is created in Rust be leveraging ggml data structures and the weights are passed into the C library.
PyTorch is only needed for reading the original weights while converting them.
PyTorch is only needed for reading the original weights while converting them.