News for hackers. Stuff that matters.
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by cma on Friday January 12, 2024 @03:05PM (#38973243)
Wouldn't that mean (until higher level embeddings) compound phrases far away are unordered? And numbers are fragmented by token boundary and scrambled up?
Re:Phi-2: Self-Extend Boosts Performance, Extends Context to... by valine on Friday January 12, 2024 @03:47PM (#38973789)
Embeddings at lower layers aren’t going to be looking very far beyond nearby or adjacent embeddings as they refine their meaning. For a number like 3.14, the tokens 3 and 14 are important to each other, but entirely unimportant to understand the meaning of a question later in the context. It’s only at later layers that an embedding representing the concept of PI becomes important to the question embeddings.

As I understand it the positional encodings are calculated relative to the token in question. It’s not like 3 and 14 are unordered tokens from their own perspective.