News for hackers. Stuff that matters.

A comment on Transformer Attention is off by one. Parent comment

Re:Transformer Attention is off by one by sebzim4500 on Monday July 24, 2023 @03:32PM (#36853143)
Are you sure? I have never seen an LLM that did not have a special token for start of text, I'm certain that llama had one and I don't remember anywhere in the llama-2 paper where they said they removed it.
[ Parent | on HN ]
Re:Transformer Attention is off by one by refulgentis on Monday July 24, 2023 @03:42PM (#36853233)
tl;dr: you're right

it's messy though, bear with me for the full explanation:

- your initial post says "<bot>" token, which looked like a mix of "chatbot" and ChatML, used by OpenAI

- there is a bo_S_ token, which acts as you described

- I averaged my attention over your post and the initial reply, which answers as if you were using "<bot>" in the misunderstood way

- when I go back and read your post, I realize the chatbot interpretation doesn't quite make sense, since you're referring to much more technical aspects than general "how do I AI", i.e. you understand <X> as a way to denote special tokens, not necessarily an XML tag

[ Parent | on HN ]