News for hackers. Stuff that matters.
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by hexaga on Friday January 12, 2024 @02:20PM (#38972622)
The core problem is: there's not enough unique, trained positions. Naively going past the end of training ctx makes you run straight into out of distribution positions, and things become incoherent.

For a model trained with a ctx size of 2, that looks like: `[0, 1, *incoherence starts* 2, 3]`

Existing RoPE scaling methods try to stay in-distribution by assigning positions between the known in-distribution ones: `[0, 0.5, 1, 1.5]`. This is still ~kinda OOD, but works w/ some fine tuning.

The method in OP breaks the core premise that we need unique positions at all, and just gives multiple tokens the same position: `[0, 0, 1, 1]`.

There are no comments on this comment yet.