News for hackers. Stuff that matters.
Re:Gemma 2: Improving Open Language Models at a Practical Si... by danielmarkbruce on Thursday June 27, 2024 @10:42AM (#40811050)
think about training.
Re:Gemma 2: Improving Open Language Models at a Practical Si... by moffkalast on Thursday June 27, 2024 @11:00AM (#40811269)
I suppose it would act as a concrete separator when instruct tuning, but lots of prompt templates don't use it, especially older ones like Alpaca. Maybe it leads to more overall coherence?
Re:Gemma 2: Improving Open Language Models at a Practical Si... by m00x on Thursday June 27, 2024 @11:06AM (#40811360)
Not instruct tuning, you use it in general training.

If you have a bunch of small prompts/answers, you can fit them into bigger batches if you use start/stop tokens.