I tried that for awhile too. Fun tip: You don't have to retrain the entire model after looping it to make it coherent again.. Just pick layers carefully then repair the residual by doing some parent distillation. ;)
That took me from needing to rent space in avast to just running it locally on my sparks and getting comparable results.
I also discovered that you can fix a lot of the damage caused by quantization the same way. Just do 200-500 rounds of parent student distillation with everything frozen but the residual, often the quantized version will improve drastically.
That took me from needing to rent space in avast to just running it locally on my sparks and getting comparable results.
I also discovered that you can fix a lot of the damage caused by quantization the same way. Just do 200-500 rounds of parent student distillation with everything frozen but the residual, often the quantized version will improve drastically.