It's interesting for localized learning... You have some local data that you want the model to incorporate, but don't want it to forget the main dataset. So you can create one of these distilled datasets and include it in a batch with the local data when you updated the model.