> I wonder if having some intrinsic randomness to the AI model would help to mitigate this problem?

This would result in identical performances resulting in different scores with the difference being artificially arbitrary.

So just like human judges?

Sure, you can reduce all of humanity to a random number generator, if you want

Maybe a slightly more generous take isn't that humans are random, but their value systems have some sort of distribution. I'm unfamiliar with gymnastics scoring, but judges may have different perspectives on what matters most if you take a sport like boxing that applies subjective scoring. One judge may value footwork and the jab more, while another weights power punches more, while another is swayed by seeing a competitor who is bloodier by the end of the fight despite having a more technically proficient fight.

Which for two identical performances - an impossibility - would be an issue, but for two different performances would still yield different scores.

I was thinking you’d be consistent within a competition, but different between competitions, so Ana, Betty and Carola face the same judges when competing in the olympics final, but their performance cannot directly be compared to Diana, Edith and Frida who completed at the world gymnastics championship.