Generative "Pre-Trained" Transformer - GPT
They did not start with a transformer that had arbitrary parameters, they started with a transformer that had been pre-trained.
Pre-training refers to unsupervised training that's done before a model is fine-tuned. The model still starts out random before it's pre-trained.
Here's where the Othello paper's weights are (randomly) initialized:
https://github.com/likenneth/othello_world/blob/master/mingp...
They didn't start with an existing model. They trained a small GPT from scratch, so the resulting model had never seen any inputs except Othello moves.