Techniques for more efficient LLM serving (up to 10x)

Posted by thimoo on Thursday May 02, 2024 @05:31PM
from the octo.ai dept.

https://octo.ai/blog/acceleration-is-all-you-need-techniques-powering-octostacks-10x-performance-boost/

( 1 comment | 5 points | on HN )

Re:Techniques for more efficient LLM serving (up to 10x) by Xaena on Thursday May 02, 2024 @06:48PM (#40242232)

Interesting benchmarks. I'm surprised vLLM is performing so poorly given all the corporate investment/attention it's getting. Is it because the contributions are more distributed than the average LLM team?

[ Parent | on HN ]