News for hackers. Stuff that matters.

A comment on Continuous batch enables 23x throughput in LLM inference and reduce p50 latency. Parent comment

Re:Continuous batch enables 23x throughput in LLM inference ... by george_123 on Tuesday August 15, 2023 @03:19PM (#37138107)
1) when the blog was released, TGI didn’t support paged attention, 2) many people don’t even know about TGI to reduce inference costs.
[ Parent | on HN ]

There are no comments on this comment yet.