Re:Llama2.rs: One-file Rust implementation of Llama2
by dietr1ch on Sunday August 06, 2023 @02:56AM
(#37019594)
I think that with C++ projects the only goal there is to keep the build simple since building C++ sucks. This is also the reason why there's many header-only libraries.
Re:Llama2.rs: One-file Rust implementation of Llama2
by tredre3 on Saturday August 05, 2023 @07:28PM
(#37017270)
> Wonder if you can mmap the binary model in without unsafe??
Some operating systems do provide the proper guarantees to make mmap safe, but Rust decided it would be best to assume it's unsafe and maintain a uniform API. Which is probably a good call, it is notoriously difficult to get a readonly mmap to be entirely safe on Linux.
Re:Llama2.rs: One-file Rust implementation of Llama2
by cchance on Sunday August 06, 2023 @12:28AM
(#37018978)
is it just me or do others hate seeing structs with fields like x, xb, xb2, hb, hb2, q, k, v with comments to describe what they are drives me crazy...
why do some dev's insist on using short names in source code, it's going to get compiled away, just use descriptive names for variables it makes it so much easier to read, it seems to be something from c/c++, as I see it mostly in code from those, most rust code tends to have much more descriptive naming.
and then comments like this...
// token embedding table
token_embedding_table: [[fx; DIM]; VOCAB_SIZE], // (vocab_size, dim)
like commenting's an artform some times, but if your comments are literally just the exact same as the following/previous text, why are you commenting it :)
Re:Llama2.rs: One-file Rust implementation of Llama2
by jokethrowaway on Sunday August 06, 2023 @03:32AM
(#37019748)
Very nice!
I wanted to do something like this but then I would miss on proper CUDA acceleration and lose performance compared to using torchlib.
I wrote a forgettable llama implementation for https://github.com/LaurentMazare/tch-rs (pytorch's torchlib rust binding).
Still not ideal but at least you get the same GPU performance you would get on pytorch.
Re:Llama2.rs: One-file Rust implementation of Llama2
by ReactiveJelly on Monday August 07, 2023 @09:24AM
(#37034115)
(since you asked for a code review)
For timing benchmarks, use Instant or a similar monotonic clock instead of SystemTime.
The original C code makes the same mistake, using clock_realtime instead of clock_monotonic.
This means the benchmarks will be wrong if the program runs while ntp is fixing up the clock. This can happen right after the system gets internet, or periodically when it checks for skew. Some systems might slowly blend in ntp fixes too, which means 1 second of calendar time is not 1 second of monotonic time over a long period of time.
At least it won't be affected by daylight saving. But it's not airtight
StackOverflow| Benefits of header-only libraries: https://stackoverflow.com/questions/12671383/benefits-of-hea...
Seeing a few uses of `unsafe`, a few of `expect`. Wonder if you can mmap the binary model in without unsafe??
Some operating systems do provide the proper guarantees to make mmap safe, but Rust decided it would be best to assume it's unsafe and maintain a uniform API. Which is probably a good call, it is notoriously difficult to get a readonly mmap to be entirely safe on Linux.
https://docs.rs/mmap-rs/latest/mmap_rs/struct.MmapOptions.ht...
If you map the file into readonly memory and than get references to it, the underlying memory can mutate (eg: by modifying the file itself).
why do some dev's insist on using short names in source code, it's going to get compiled away, just use descriptive names for variables it makes it so much easier to read, it seems to be something from c/c++, as I see it mostly in code from those, most rust code tends to have much more descriptive naming.
and then comments like this... // token embedding table token_embedding_table: [[fx; DIM]; VOCAB_SIZE], // (vocab_size, dim)
like commenting's an artform some times, but if your comments are literally just the exact same as the following/previous text, why are you commenting it :)
like the file formats, all the extra files like the tokenizer.bin file, the terminology in the sources comments, logits, transformers etc
I wrote a forgettable llama implementation for https://github.com/LaurentMazare/tch-rs (pytorch's torchlib rust binding). Still not ideal but at least you get the same GPU performance you would get on pytorch.
...And then I spotted Candle, a new ML framework by the same author: https://github.com/huggingface/candle
It's all in Rust, self contained, a huge undertaking, but it looks very promising. They already have a llama2 example!
For timing benchmarks, use Instant or a similar monotonic clock instead of SystemTime.
The original C code makes the same mistake, using clock_realtime instead of clock_monotonic.
This means the benchmarks will be wrong if the program runs while ntp is fixing up the clock. This can happen right after the system gets internet, or periodically when it checks for skew. Some systems might slowly blend in ntp fixes too, which means 1 second of calendar time is not 1 second of monotonic time over a long period of time.
At least it won't be affected by daylight saving. But it's not airtight