Render a neural network into CUDA/HIP code

Re:Render a neural network into CUDA/HIP code by antinucleon on Friday June 02, 2023 @02:37PM (#36169032)

Hippo is faster than AITemplate, and supports more generative models. We haven't compared vs TVM, but for absolute token/s on M2 Max, Hippo is able to running decoding on LLAMA with datacenter level GPUs (with other SW).

[ Parent | on HN ]

Re:Render a neural network into CUDA/HIP code by huevosabio on Friday June 02, 2023 @03:24PM (#36169627)

Thanks, I've added myself to the waitlist. Please let us know when this can be tried!

[ Parent | on HN ]

Re:Render a neural network into CUDA/HIP code by choppaface on Friday June 02, 2023 @08:31PM (#36172538)

How does Hippo compare with TensorRT?

[ Parent | on HN ]

Re:Render a neural network into CUDA/HIP code by philipturner on Saturday June 03, 2023 @11:45PM (#36182908)

The difference in bandwidth between M2 Max and data centers GPUs isn't that much (less than a factor of 5). The difference in compute is much, much larger. If you only have fast GEMV kernels, and not fast GEMM kernels, you're basically locked into an inference engine that can only run GPT-style transformers efficiently. However, it can *technically* support all the models, but at what ALU utilization?

[ Parent | on HN ]