Re:Using mmap to make LLaMA load faster
by astrange on Wednesday April 05, 2023 @01:18PM
(#35457448)
> while some of the most basic optimizations are seemingly a lost art
mmap isn't relevant to anyone except CPU-using programmers because other hardware doesn't have virtual memory paging. Firmware programmers don't care, GPU programmers don't care.
AFAIK CUDA offers unified memory which basically works with virtual address space and page faulting in data from main memory. There is also IOMMU in general.
Many of us would like to get rid of the host CPU and have ML trainers that are just GPUs and drives and NICs all attached to a northbridge. The GPU has everything required to make disk requests over the bus, and ideally the drive can receive network messages that get plumbed straight to the drive (I'm only partially joking).
Re:Using mmap to make LLaMA load faster
by telotortium on Wednesday April 05, 2023 @01:11PM
(#35457337)
Probably a combination of (a) ML framework people not paying much attention to CPU inference due to already having GPUs/TPUs already lying around for training - CPU inference is just for very quick experiments (b) research code has never been the best optimized for performance (c) ML people are not generally systems programmers, and a lot of systems programmers are afraid to mess with the ML code outside of low-level computation kernels (doesn't help that ML code is notoriously unreproducible).
It's indeed a very different world. This model was trained on thousands of GPUs. The weird file format corresponds to the train time sharding of the weights. And really nobody is doing CPU inference with all the GPU we have. And also the "CLI" use case seems contrieved to me. If you plan to interact several times with the model and want to keep the weights in RAM, why don't you start a REPL or spin up a server?
Re:Using mmap to make LLaMA load faster
by kristjansson on Wednesday April 05, 2023 @01:10PM
(#35457313)
> never optimized to begin with
I think the better read is that they're being adapted to new applications, constraints, and environments, all at once.
Re:Using mmap to make LLaMA load faster
by danieldk on Wednesday April 05, 2023 @02:51PM
(#35458790)
Word embeddings were big for their time (especially with subword embeddings like fastText). We mmaped word embeddings for similar reasons. But yeah, I was kinda surprised that one post about LLaMa.cpp mmap support talked about a 'fairly new technique'. mmap has been in a UNIX programmer's tool belt for literally decades.
On the other hand, its only been a few weeks, so maybe I should ignore this absurdity and just wait.