vLLM: 24x faster LLM serving than HuggingFace Transformers

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by karmasimida on Tuesday June 20, 2023 @07:28PM (#36412208)

I think they are orthogonal.

Flash attention is just another way to compute exact attention.

This work mainly concerns how to resolve memory fragmentation across different sequences

You still need to compute attention as is once you retrieve the needed key values

[ Parent | on HN ]

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by wskwon on Tuesday June 20, 2023 @08:13PM (#36412592)

Thanks for the explanation! I believe the two ideas are basically orthogonal. FlashAttention reduces memory read/writes, while PagedAttention reduces memory waste.

[ Parent | on HN ]