News for hackers. Stuff that matters.
Re:Punica: Serving multiple LoRA finetuned LLM as one by zhye on Wednesday November 08, 2023 @06:43PM (#38198881)
Good question, in general implementing kernels on page tables is tricky in Tensor Compilers because integer set analysis might fail sometimes (but can be fixed with some tweaks). I think using compilers like TVM can help deploy serving systems on different platforms (e.g. AMD GPUs) and I'm optimistic about this direction (and we have to make Tensor Compilers more user-friendly).

There are no comments on this comment yet.