Re:Gemma 2: Improving Open Language Models at a Practical Si...
by alekandreev on Thursday June 27, 2024 @11:03AM
(#40811310)
Your training input has the shape of (sequence length x batch size). If a lot of your samples are shorter than sequence length, as is usually the case, you will have a lot of padding tokens in the input, which is wasted compute.
To compensate for that, you can pack multiple examples in the same sequence. This is there EOS and BOS come in, as they indicate to the model that the two parts of the sequence are not related.
Re:Gemma 2: Improving Open Language Models at a Practical Si...
by thomasahle on Thursday June 27, 2024 @11:25AM
(#40811559)
You can just do that my shaping the attention mask, no?
That also gives you an actual guarantee that no information is leaked between conversations.
Re:Gemma 2: Improving Open Language Models at a Practical Si...
by suryabhupa on Thursday June 27, 2024 @12:09PM
(#40812058)
In practice, and at scale, that's exactly what having <bos> and <eos> tokens allow you to easily and programmatically do.
Re:Gemma 2: Improving Open Language Models at a Practical Si...
by danielmarkbruce on Thursday June 27, 2024 @04:52PM
(#40814963)
You can't pack multiple examples into a single row of a matrix without knowing where one begins and one ends.
Re:Gemma 2: Improving Open Language Models at a Practical Si...
by danielmarkbruce on Thursday June 27, 2024 @10:42AM
(#40811050)
think about training.
Re:Gemma 2: Improving Open Language Models at a Practical Si...
by moffkalast on Thursday June 27, 2024 @11:00AM
(#40811269)
I suppose it would act as a concrete separator when instruct tuning, but lots of prompt templates don't use it, especially older ones like Alpaca. Maybe it leads to more overall coherence?
> User turn: user
> Model turn: model
> Start of conversation turn: <start_of_turn>
> End of conversation turn: <end_of_turn>
> Beginning of sequence: <bos>
> End of sequence: <eos>
You know I keep wondering why <bos> and <eos> tokens are even a thing in general. No model is tuned to keep generating multiple turns after its <end_of_turn> equivalent is sent, and what's the point of <bos> when you're parsing the entire context anyway. If it's an attempt to ignore text before it... then why is that text there? Just remove it from context, you're throwing away compute.