distilbert was trained from Bert. there might be an angle using another model to train the model especially if your trying to get something to run locally.