Re:Transformer Attention is off by one
by refulgentis on Monday July 24, 2023 @03:42PM
(#36853233)
tl;dr: you're right
it's messy though, bear with me for the full explanation:
- your initial post says "<bot>" token, which looked like a mix of "chatbot" and ChatML, used by OpenAI
- there is a bo_S_ token, which acts as you described
- I averaged my attention over your post and the initial reply, which answers as if you were using "<bot>" in the misunderstood way
- when I go back and read your post, I realize the chatbot interpretation doesn't quite make sense, since you're referring to much more technical aspects than general "how do I AI", i.e. you understand <X> as a way to denote special tokens, not necessarily an XML tag
it's messy though, bear with me for the full explanation:
- your initial post says "<bot>" token, which looked like a mix of "chatbot" and ChatML, used by OpenAI
- there is a bo_S_ token, which acts as you described
- I averaged my attention over your post and the initial reply, which answers as if you were using "<bot>" in the misunderstood way
- when I go back and read your post, I realize the chatbot interpretation doesn't quite make sense, since you're referring to much more technical aspects than general "how do I AI", i.e. you understand <X> as a way to denote special tokens, not necessarily an XML tag