vLLM: 24x faster LLM serving than HuggingFace Transformers

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by thomasahle on Tuesday June 20, 2023 @07:17PM (#36412091)

I wonder how this compares to Flash Attention (https://github.com/HazyResearch/flash-attention), which is the other "memory aware" Attention project I'm aware of.

I guess Flash Attention is more about utilizing memory GPU SRam correctly, where this is more about using the OS/CPU memory better?

[ Parent | on HN ]

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by karmasimida on Tuesday June 20, 2023 @07:28PM (#36412208)

I think they are orthogonal.

Flash attention is just another way to compute exact attention.

This work mainly concerns how to resolve memory fragmentation across different sequences

You still need to compute attention as is once you retrieve the needed key values

[ Parent | on HN ]

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by wskwon on Tuesday June 20, 2023 @08:13PM (#36412592)

Thanks for the explanation! I believe the two ideas are basically orthogonal. FlashAttention reduces memory read/writes, while PagedAttention reduces memory waste.

[ Parent | on HN ]

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by ipsum2 on Tuesday June 20, 2023 @07:26PM (#36412188)

The ideas are orthogonal, and can be used (theoretically) at the same time.

[ Parent | on HN ]

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by scv119 on Tuesday June 20, 2023 @07:48PM (#36412384)

I believe you can slightly change the flash attention kernel to implement the same kernel of this page attention, since both of them work on the key/value cache at block level.

[ Parent | on HN ]