Render a neural network into CUDA/HIP code

Re:Render a neural network into CUDA/HIP code by boywitharupee on Friday June 02, 2023 @03:01PM (#36169316)

zero-copy with mmap was added to llama.cpp, but the way it was implemented sparked controversy.

I'm particularly interested in a specific aspect of the problem. Do you load weights with zero-copy using Metal? MTLIOCommandBuffer offers convenient functions to directly read a file into a MTLBuffer. Which was part of Metal 3 only.

[ Parent | on HN ]

Re:Render a neural network into CUDA/HIP code by fathyb on Friday June 02, 2023 @04:02PM (#36170039)

I think GP meant zero-copy communication with the GPU, eg. through `newBufferWithBytesNoCopy` [0], which is only possible with unified memory architectures, eg. integrated GPUs.

The mmap change was just about mapping the model files in memory instead of copying them, which has less overhead.

[0]: https://developer.apple.com/documentation/metal/mtldevice/14...

[ Parent | on HN ]

Re:Render a neural network into CUDA/HIP code by brucethemoose2 on Friday June 02, 2023 @08:42PM (#36172595)

Yeah this is precisely what I meant. I think it is possible in Vulkan too.

I brought it up because shuffling weights around takes lots of time, takes up more RAM, and saps memory bandwidth the IGP and CPU desparately need.

[ Parent | on HN ]

Re:Render a neural network into CUDA/HIP code by astrange on Saturday June 03, 2023 @02:36AM (#36174288)

> I think GP meant zero-copy communication with the GPU, eg. through `newBufferWithBytesNoCopy` [0], which is only possible with unified memory architectures, eg. integrated GPUs.

It can still be beneficial on discrete GPUs because it avoids a copy inside the driver.

[ Parent | on HN ]