I plan on checking out RWKV and seeing if I can add my sacrifical training techniques to it this weekend. There is a reason quantization works, it is because models are very badly trained right now. I think we can get really good performance on .1b and 1b models which opens up the world to fine-tuning again. I was playing with fine-tuning llama 7b and 13b a while back but the HW/SW stack made it so unwieldy and the ROI was terrible compared to just adjusting prompts on gpt-4o-mini and the like. I have hope that we are about to see single GPU, very simple, fine-tuning again as models shrink and GPUs grow.