News for hackers. Stuff that matters.

A comment on Llm.c – LLM training in simple, pure C/CUDA.

Re:Llm.c – LLM training in simple, pure C/CUDA by convexstrictly on Monday April 08, 2024 @10:40PM (#39975812)
Candle is a minimalist ML framework for Rust with a focus on performance (including GPU support) and ease of use

https://github.com/huggingface/candle

[ Parent | on HN ]
Re:Llm.c – LLM training in simple, pure C/CUDA by imjonse on Tuesday April 09, 2024 @12:33AM (#39976215)
Candle focuses on inference though.
[ Parent | on HN ]
Re:Llm.c – LLM training in simple, pure C/CUDA by l-m-z on Tuesday April 09, 2024 @01:48AM (#39976485)
Candle dev here, we also support training/backdrop! We certainly focus on optimizing inference performance but hopefully that should improve the training efficiency too.
[ Parent | on HN ]
Re:Llm.c – LLM training in simple, pure C/CUDA by revskill on Tuesday April 09, 2024 @06:09AM (#39977883)
What is referecing ?
[ Parent | on HN ]
Re:Llm.c – LLM training in simple, pure C/CUDA by HarHarVeryFunny on Tuesday April 09, 2024 @11:25AM (#39980465)
Inference means using the neural net, as opposed to training it.

During inference you feed an input into the NN and it passes through it in "forwards" direction (i.e. from input to output), being modified according to the "weights" that were learnt during training, to derive the output.

During training, each training sample is first fed forwards through the NN, the same way as for inference, but then the output of the model (which at the beginning of training will be random/wrong) is compared to the correct/desired output for that training sample, and a corresponding error value will then be fed backwards (from output to input) through the NN according to the "backpropagation" mechanism to update the weights.

Training is a lot more involved than inference since it involves this backpropagation step.

[ Parent | on HN ]
Re:Llm.c – LLM training in simple, pure C/CUDA by basbuller on Tuesday April 09, 2024 @01:14AM (#39976333)
Not barely as minimal as Karpathy his implementation
[ Parent | on HN ]
Re:Llm.c – LLM training in simple, pure C/CUDA by jeroenvlek on Tuesday April 09, 2024 @02:12AM (#39976576)
Love Candle! I actually ported Karpathy's previous GPT tutorial to candle, including training [0]

[0] https://www.perceptivebits.com/building-gpt-from-scratch-in-...

[ Parent | on HN ]
Re:Llm.c – LLM training in simple, pure C/CUDA by 0xfedbee on Tuesday April 09, 2024 @03:44AM (#39977036)
I wouldn't call in "minimalist" after seeing Karpathy's code.
[ Parent | on HN ]
Re:Llm.c – LLM training in simple, pure C/CUDA by r-k-jo on Thursday April 11, 2024 @02:00AM (#39998846)
You can also target WASM. Depending on the model size, it can be really good. Here are some examples of quantized models.

Vision Model https://huggingface.co/spaces/radames/Candle-Moondream-2 Blip Image Captioning https://huggingface.co/spaces/radames/Candle-BLIP-Image-Capt... Microsoft Phi 2 https://huggingface.co/spaces/radames/Candle-phi1-phi2-wasm-...

[ Parent | on HN ]