Re:Reformer, the Efficient Transformer
by marcinzm on Sunday January 19, 2020 @06:55PM
(#22095054)
Wouldn't the important embedding be of the token at position A and the token at position B rather than the positions themselves? Since there's a lot fewer tokens than positions you're fairly likely to get the two tokens to hash together at least a few times.
edit: I'd image the position itself (ie: word number in text rather than token) could be embedded using sin-cosine or by breaking it up into chapter/paragraph/word. Seems more meaningful and efficient than word number in text. That would prevent this issue on that side of things.
edit: I'd image the position itself (ie: word number in text rather than token) could be embedded using sin-cosine or by breaking it up into chapter/paragraph/word. Seems more meaningful and efficient than word number in text. That would prevent this issue on that side of things.