I think there was a paper from Google showing that if you included 5% of your original dataset together with the new data during the finetuning then catastrophical forgetting didn't occur. Perhaps it's that simple.