Re:There Will Be a Scientific Theory of Deep Learning
by imtringued on Saturday April 25, 2026 @05:29AM
(#47900028)
Ok but it's already known that you shouldn't initialize your network parameters to a single constant and instead initialize the parameters with random numbers.
Re:There Will Be a Scientific Theory of Deep Learning
by charcircuit on Saturday April 25, 2026 @06:35AM
(#47900342)
The model can converge towards such a state even if randomly initialized.
Re:There Will Be a Scientific Theory of Deep Learning
by hodgehog11 on Saturday April 25, 2026 @08:16AM
(#47900830)
Both you and the comment above are correct; initializing with iid elements ensures that correlations are not disastrous for training, but strong correlations are baked into the weights during training, so pretty much anything could potentially happen.
If they were all correlated with each other that does not seem far fetched.