News for hackers. Stuff that matters.
Re:Our Humble Attempt at “How Much Data Is Needed to Fine-Tune by BoorishBears on Monday September 25, 2023 @07:01PM (#37651942)
Putting knowledge in the context window enables the same thing

For example, letting the LLM return a result to the user and return a list of entities it wants to save details on for later, then letting it query past entries allows long term memory for a class of entities.

Over all it will be a very long time before approaches like that are beat by fine tuning, because as the models improve, this strategy gets stronger: the model gets better are crafting queries, you can provide more context in the context window.

Meanwhile fine tuning in knowledge will get more difficult as MOE and other unforseen developments reset what works as far as directly tinkering with weights.

There are no comments on this comment yet.