Attention with Linear Biases (ALiBi)

Re:Attention with Linear Biases (ALiBi) by Eridrus on Sunday May 14, 2023 @11:06PM (#35943482)

Is ALiBi still the sota for this setting, or have there been advances beyond this in the last 8 months? I know there has been a lot of interest in longer context lengths recently.

[ Parent | on HN ]

Re:Attention with Linear Biases (ALiBi) by ipsum2 on Tuesday May 16, 2023 @02:59AM (#35958525)

xpos is SoTA right now: https://arxiv.org/pdf/2212.10554.pdf

[ Parent | on HN ]

Re:Attention with Linear Biases (ALiBi) by Eridrus on Wednesday May 17, 2023 @07:02AM (#35973638)

Thanks!

[ Parent | on HN ]