That's the thing, if your dataset is not similar to the dataset on which your model was pre-trained (which it often isn't), the initial weights may actually prove to be either worthless, or even detrimental. That is, they may be worse than random initialization.