News for hackers. Stuff that matters.

A comment on Using mmap to make LLaMA load faster. Parent comment

Re:Using mmap to make LLaMA load faster by astrange on Wednesday April 05, 2023 @01:18PM (#35457448)
> while some of the most basic optimizations are seemingly a lost art

mmap isn't relevant to anyone except CPU-using programmers because other hardware doesn't have virtual memory paging. Firmware programmers don't care, GPU programmers don't care.

[ Parent | on HN ]
Re:Using mmap to make LLaMA load faster by leni536 on Wednesday April 05, 2023 @03:16PM (#35459074)
AFAIK CUDA offers unified memory which basically works with virtual address space and page faulting in data from main memory. There is also IOMMU in general.
[ Parent | on HN ]
Re:Using mmap to make LLaMA load faster by dekhn on Wednesday April 05, 2023 @06:39PM (#35461434)
Many of us would like to get rid of the host CPU and have ML trainers that are just GPUs and drives and NICs all attached to a northbridge. The GPU has everything required to make disk requests over the bus, and ideally the drive can receive network messages that get plumbed straight to the drive (I'm only partially joking).
[ Parent | on HN ]