News for hackers. Stuff that matters.

A comment on Transformer Attention is off by one. Parent comment

Re:Transformer Attention is off by one by sp332 on Monday July 24, 2023 @03:04PM (#36852838)
Chat-tuned ones do, but the base models don't. For example, Llama doesn't, but Alpaca has "### Instruction:", "### Input:", and "### Response:".
[ Parent | on HN ]
Re:Transformer Attention is off by one by int_19h on Monday July 24, 2023 @03:19PM (#36852983)
Base LLaMA still has dedicated tokens for beginning/end of string. What you're describing is the instruction format, which is separate.
[ Parent | on HN ]
Re:Transformer Attention is off by one by sp332 on Monday July 24, 2023 @03:30PM (#36853123)
Oh, I had misunderstood something.
[ Parent | on HN ]