Re:Punica: Serving multiple LoRA finetuned LLM as one
by yyding on Wednesday November 08, 2023 @04:23PM
(#38197263)
Good job!
I observed that you implemented many cuda kernels by yourselves. Just wondering your consideration or trade-off between implementating the kernels via pure CUDA code vs. implementing based on compiler like TVM/Triton.
[ Parent
| on HN ]
Re:Punica: Serving multiple LoRA finetuned LLM as one
by zhye on Wednesday November 08, 2023 @06:43PM
(#38198881)
Good question, in general implementing kernels on page tables is tricky in Tensor Compilers because integer set analysis might fail sometimes (but can be fixed with some tweaks). I think using compilers like TVM can help deploy serving systems on different platforms (e.g. AMD GPUs) and I'm optimistic about this direction (and we have to make Tensor Compilers more user-friendly).