News for hackers. Stuff that matters.
Re:Llama 32K Context Released by Together AI by npsomaratna on Saturday July 29, 2023 @01:41AM (#36917284)
My understanding is that in NTK aware RoPE scaling, the model does pay uniform attention. With older methods, not as much.

There are no comments on this comment yet.