Re:Self-reasoning tokens: teaching models to think ahead
by PeterisP on Saturday April 20, 2024 @05:25PM
(#40101164)
> In my specific case it was the bias to provide an answer to a question
That seems to be a reasonably expected result of the "instruction post-training" finetuning with RLHF or otherwise. If for some reason you don't want this behavior, you can avoid this by using a model version that just has the core language modeling without that finetuning, e.g. the llama models have such a version available.
Re:Self-reasoning tokens: teaching models to think ahead
by XenophileJKO on Saturday April 20, 2024 @08:02PM
(#40102100)
Well in this specific case, the logic I was asking the model to do was. (Highly paraphrased..)
1. Inventory the retrieved items.
2. Determine their relevance.
3. Pick the most relevant or if none of the retrieved items is relevant return an alternative message.
What the model will do is add new items into (1) if none of the retrieved items are relevant. If you add some steps between 1 and 2.. it stops doing that.
Re:Self-reasoning tokens: teaching models to think ahead
by alt0_ on Saturday April 20, 2024 @06:41PM
(#40101658)
> definately
relevant xkcd: https://xkcd.com/2871/
Re:Self-reasoning tokens: teaching models to think ahead
by XenophileJKO on Saturday April 20, 2024 @07:59PM
(#40102080)
If I wanted it spelled correctly I would have run it through the LLM.
Basically it will create pre-conditions so that the final output aligns to some bias. In my specific case it was the bias to provide an answer to a question. This is noticable sometimes in chain of thought intermediate outputs. I ended up having to create some space between the entangled decisions in the chain of thought output.