Re:vLLM: 24x faster LLM serving than HuggingFace Transformers
by scv119 on Tuesday June 20, 2023 @07:48PM
(#36412384)
I believe you can slightly change the flash attention kernel to implement the same kernel of this page attention, since both of them work on the key/value cache at block level.
[ Parent
| on HN ]
There are no comments on this comment yet.