From what I can tell, this doesn’t re-seed the random number generator, so we shouldn’t expect deterministic results. A better test would be to examine the model’s logits, or probabilities of the next token, across different ram sizes.