Re:Alpaca: A Strong Open-Source Instruction-Following Model
by leodriesch on Tuesday March 14, 2023 @09:20AM
(#35150998)
They show an example of this under "Training Recipe". They prompt Davinci to give them instruction following examples, 20 at a time. And that a lot of times. The context window is also 4000 tokens, at least that is the maximum amount you can pick in the OpenAI playground.
What I don't understand is how did they achieve giving this context to GPT, since afaik the context window is just 2048 tokens? It seems that it's not fine-tuning, but few-shot.