News for hackers. Stuff that matters.

A comment on GPT-4 Details Leaked. Parent comment

Re:GPT-4 Details Leaked by why_only_15 on Tuesday July 11, 2023 @07:39PM (#36688637)
presumably you mean a dot product of Q and K, and no you do not have to store this: https://arxiv.org/abs/2205.14135
[ Parent | on HN ]
Re:GPT-4 Details Leaked by RC_ITR on Wednesday July 12, 2023 @12:34PM (#36697221)
I mean, sure you can work around it, but from your own link:

>since the time and memory complexity of self-attention are quadratic in sequence length

[ Parent | on HN ]
Re:GPT-4 Details Leaked by why_only_15 on Wednesday July 12, 2023 @01:37PM (#36698468)
Except in practice this is not true, and hasn't been for more than a year. It's not just a workaround either -- FlashAttention is both faster at runtime and uses less memory.
[ Parent | on HN ]