News for hackers. Stuff that matters.

A comment on vLLM: 24x faster LLM serving than HuggingFace Transformers. Parent comment

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by ipsum2 on Tuesday June 20, 2023 @07:26PM (#36412188)
The ideas are orthogonal, and can be used (theoretically) at the same time.
[ Parent | on HN ]
Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by scv119 on Tuesday June 20, 2023 @07:48PM (#36412384)
I believe you can slightly change the flash attention kernel to implement the same kernel of this page attention, since both of them work on the key/value cache at block level.
[ Parent | on HN ]