Not this one, but Google's PaLM (which is 4x OPT3) trains semi-deterministically.

These kinds of large transformers can be relatively reproduceable in results and benchmarks. However, making them converge to the exact same parameter set might not be a reasonable expectation.

> how reproducible is the final output model?

Not much, but it also depends on what you mean by "reproducible".

Do you expect a similar internal representation (weights) or a similar behavior?

the details of that specific model they ended up with? Irreproducible, unless the system was carefully designed and every detail required to do a fully reproducible computation was recorded and replayed. But they could easily produce a bunch of models that all sort of end up in roughly the same place and perform the same, ideally reducing the number of things they needed to change ad-hoc during the training.