Notes on Weight Initialization for Deep Neural Networks

Re:Notes on Weight Initialization for Deep Neural Networks by enriquto on Tuesday May 21, 2019 @06:48AM (#19968972)

> We can divide by a number (scaling_factor) to scale down its magnitude to the right level

This argument bugs me a bit... since these numbers are represented using floating point, whose precision does not depend on their magnitude, what is the point of scaling them?

Furthermore, I do not believe his first example. Is torch really that bad? In octave:

    x = randn(512, 1);
    A = randn(512);
    y = A^100 * x;
    mean(p), std(p)

gives regular numbers ( 9.1118e+135 and 1.9190e+137 )

They are large, but far from overflowing. And this corresponds to a network of deep 100, which is not a realistic scenario.

[ Parent | on HN ]

Re:Notes on Weight Initialization for Deep Neural Networks by KidComputer on Tuesday May 21, 2019 @10:12AM (#19970311)

That's because octave is using doubles. You can do the exact same thing in PyTorch by passing in dtype=torch.float64 into torch.randn.

[ Parent | on HN ]

Re:Notes on Weight Initialization for Deep Neural Networks by axiom92 on Tuesday May 21, 2019 @02:01PM (#19972840)

> They are large, but far from overflowing.

Sure, but isn't large relative? Sure you can make them overflow in octave as well, given enough layers. Which brings us to next point :-)

> And this corresponds to a network of deep 100, which is not a realistic scenario.

Actually deep 100 is not unrealistic at all these days! https://arxiv.org/abs/1611.09326

[ Parent | on HN ]

Re:Notes on Weight Initialization for Deep Neural Networks by L2R on Wednesday May 22, 2019 @01:19PM (#19983271)

There are approaches to ensure parameters remain stable despite the depth (selu, for example).

[ Parent | on HN ]