The training sources are:
1. Google Synthetic PersonaChat
2. DailyDialog
3. SODA conversational data
4. Hand-written model-personality conversations, repeated during training
5. Identity-corrected conversations—rewritten names, human-life claims, and hobby responses
6. 960 synthetic recovery conversations, teaching it to admit ignorance after user push-back
The final set contains 23,270 conversations
However, the chain reasoning and math problem solving skills, I was only able to add recently, by grafting certain parts from a 1B model, then doing space-weight programming, and finally finetuning/training to get the model to "get used to" reasoning
The full model has
* Frozen base model to interpret questions
* Additional trained heads to select question types, evidence, and follow-up behavior
* A library tool for retrieving facts
* Working memory and internal script language execution for arithmetic, conversions, and comparisons across answers. For example, it can retrieve a building’s height, remember that number, then execute “half of that”
* Identity extraction and physical simulation, so you can do thing like: "Ben has the ball", "Who has the ball", Model: "Ben", "Ben gave the ball to Sally", "Who has the ball now?", "Sally"
It's relatively easy adding skills either via its internal scripting language, grafting+weight programming, fine-tuning and/or training it to use a new skill. The main limitation right now seems to be just quality data and proper supervision, ie. someone to talk to the model flag issues, fix them, then re-test. This is kinda tedious, but I've found that even the best frontier model is kinda lacking in quality when producing synthetic conversational data, and most datasets out there have a lot of cruft. So I get better results just manually producing the data and iterating, than feeding it GBs of datasets
Creating and training this model has been a lot of fun. Since it's so tiny, it's easy to experiment with and try a lot of different things. Most training I can just do in a few minutes locally