Re:Lossless Acceleration of LLM via Adaptive N-Gram Parallel...
by SonOfLilit on Monday April 22, 2024 @12:00PM
(#40115613)
As I said, the attention step is O(n^2) per token sequentially and O(n^2) for the entire sequence when calculating the entire sequence in parallel, where n is the length of the sequence.
The main reason I can come up with why doing the same calculation 8 times in parallel instead of 8 times sequentially is that you benefit from better locality of reference.