News for hackers. Stuff that matters.
Re:Alpaca: A Strong Open-Source Instruction-Following Model by frozenwind on Tuesday March 14, 2023 @04:52AM (#35148838)
From what I understood they used 175 seed tasks as an example for da-vinci which then generated the 52K instructions used to fine-tune LLama.

What I don't understand is how did they achieve giving this context to GPT, since afaik the context window is just 2048 tokens? It seems that it's not fine-tuning, but few-shot.

Re:Alpaca: A Strong Open-Source Instruction-Following Model by leodriesch on Tuesday March 14, 2023 @09:20AM (#35150998)
They show an example of this under "Training Recipe". They prompt Davinci to give them instruction following examples, 20 at a time. And that a lot of times. The context window is also 4000 tokens, at least that is the maximum amount you can pick in the OpenAI playground.