Re:Phi-2: Self-Extend Boosts Performance, Extends Context to...
by valine on Friday January 12, 2024 @03:47PM
(#38973789)
Embeddings at lower layers aren’t going to be looking very far beyond nearby or adjacent embeddings as they refine their meaning. For a number like 3.14, the tokens 3 and 14 are important to each other, but entirely unimportant to understand the meaning of a question later in the context. It’s only at later layers that an embedding representing the concept of PI becomes important to the question embeddings.
As I understand it the positional encodings are calculated relative to the token in question. It’s not like 3 and 14 are unordered tokens from their own perspective.
As I understand it the positional encodings are calculated relative to the token in question. It’s not like 3 and 14 are unordered tokens from their own perspective.