1. Requires fine-tuning every 10M tokens ingested.

2. Requires complicated multi model pipelines. One for distilling the context, one for making a training dataset, and one for applying it.

3. All of this requires a lot of compute. Note the lack of wall times in the paper. While they do note that their process is considerably faster than SFT they never really quantify it.

4. They only published results on Llama and Qwen, and on fairly small versions to boot.

Despite all of that, the paper is a starting point for a process of making a model that can modify its own weights. I very much suspect that Google has solved many of the problems given the breadth of the related work and additional experiments sections in the appendix.

It will be really fun to see open weight models come out in the future with the functionality described in the paper.