There's some super-interesting recent empirical work from MS research that makes this perspective less likely to be a useful one. In short, the behavior of neural networks in a few regimes makes this interpretation of them less likely, specifically: ensembling the same network trained multiple times separately improves results compared to training multiple networks concurrently, knowledge distillation works, and self-distillation works. The blog post[1] has a good intuitive explanation of the matter and a link to the paper.

[1] https://www.microsoft.com/en-us/research/blog/three-mysterie...