(Original author here) - here was a really hard-to-find bug - in some instances more than one quantity of fish could be placed in the same location - that meant the game would work fine until someone fished that same fish the second time and the fishing boats would diverge in the different simulations. The world sync check only counted the tile contents so we didn't see it. - There were actually two RNG's in the game, one was synchronized with the same start seed on all machines (basically the same random pool) for combat and whatnot - the other was unsynchronized and used for animation variance, etc -things that weren't gameplay related. Not knowing when to use one of these specifically (e.g. animals facing seems like animation but definitely affected gameplay if they could be hunted) could alter the code path and cause an out-of-sync condition.

Why even keep the unsynchronized RNG? I'm guessing it was more performant when used appropriately?

If you don't have an unsynchronized RNG, then any unsynchronized content can't use an RNG, and unsynchronized content is important for improving a game's experience by using local resources for things that don't really need to be shared or sent over a network.

For example, maybe in a FPS, part of the non-gameplay-critical graphics use particle generators for a cool effect that not all players see (because it's behind a building for some of them and thus doesn't even need to be rendered); if these generators used a synchronized RNG, then all players would have to do computations for every particle effect happening anywhere, just so that the combat and more game-important RNG values would be in synch when they really need to be.

There were a lot of extra checks and code involved in the synchronized RNG - the results were loggable (to track down sync failures) for instance so bloating that up with every random fire flicker from a burning castle would have been crazy.

I wonder if that’s something a type system (or Hungarian notation!) could have helped with.

You might want something randomized in animation -- and you might therefore only run it if something is onscreen.

If you're using a global random pool, you've descyned here unless all players have the same thing on screen.

What they mean is that if you're doing anything that samples random, don't do it inside of something like a loop whose length is dependent on the local game state that isn't sync'd on the current turn; especially in the graphics engine for doing particle effects or picking random animation frames. Like if you sample random every animation update frame to pick which fire graphic to use on a torch or bonfire (very common), but you suffer graphic slowdown and sample it not enough or too many times, that could vary between clients unintentionally.

Once it comes time to simulate that next turn, if you have something different than other clients because of a missed update or graphics lag, even if object positions and the random seed going to be "fixed" by another turn update, all future interactions with any objects that over- or under-sampled random will be wrong and could create further sync problems.

For instance, anything that's based on the current user's window rather than global state.

Ah yep, that makes sense. I wasn't thinking in terms of what had to happen for actual gameplay, just simulation state.