The "functional equivalence" discussed in the past gets at some of this. There's definitely more going on than permutation symmetry; in particular, if two solutions are symmetrically related (and thus evaluating the same overall function) then ensembling them together shouldn't help. But ensembling /does/ still help in many cases.

The different solutions found in different runs likely share a lot of information, but learn some different things on the edges. It would be cool to isolate the difference between two networks...