Re:Lumos: Learning Agents with Unified Data, Modular Design,...
by lmeyerov on Tuesday April 02, 2024 @06:21AM
(#39904093)
A few bits we've learned from building louie over the last year+:
* Training on the internet means fault tolerance out-of-the-box
* Rule-of-thumb: Sub-gpt4 models are derpy, while gpt4/claude are reasonable. The dumber the model, the more it gets tripped up on basics and the more you need to engineer.
* In areas it's already confused, prompt errors can push it over the edge: When asking for JSON, will it output JSON strings, or Python literals?
* Even GPT4 grade will get tripped up on things like negations. So simpler grammar helps. As does emphasizing where to pay attention. REALLY.
* They're simple-minded. So asking GPT4 to do multiple things, especially over a long duration, gets weird. (Long context is a bit of a myth.)
* They're sensitive to persona. Instructing it to answer carefully as a professional would helps improve quality. The opposite is less clear - the base model is just ok, and typos seem to be similarly ok.
* Agents can easily 2-100X quality, so after basics in the prompt, better to focus on the code outside the prompt.
* Training on the internet means fault tolerance out-of-the-box
* Rule-of-thumb: Sub-gpt4 models are derpy, while gpt4/claude are reasonable. The dumber the model, the more it gets tripped up on basics and the more you need to engineer.
* In areas it's already confused, prompt errors can push it over the edge: When asking for JSON, will it output JSON strings, or Python literals?
* Even GPT4 grade will get tripped up on things like negations. So simpler grammar helps. As does emphasizing where to pay attention. REALLY.
* They're simple-minded. So asking GPT4 to do multiple things, especially over a long duration, gets weird. (Long context is a bit of a myth.)
* They're sensitive to persona. Instructing it to answer carefully as a professional would helps improve quality. The opposite is less clear - the base model is just ok, and typos seem to be similarly ok.
* Agents can easily 2-100X quality, so after basics in the prompt, better to focus on the code outside the prompt.