Pre-training refers to unsupervised training that's done before a model is fine-tuned. The model still starts out random before it's pre-trained.
Here's where the Othello paper's weights are (randomly) initialized:
https://github.com/likenneth/othello_world/blob/master/mingp...
They did not start with a transformer that had arbitrary parameters, they started with a transformer that had been pre-trained.