You could still optimize the prototypes, so fine-tuning with this in place would be possible (see, e.g., [1]). But we don't yet have data on how well this would work using our exact method, how early in training you could do the op replacement, etc.

[1] http://openaccess.thecvf.com/content_ECCV_2018/html/Sanghyun...