News for hackers. Stuff that matters.

A comment on Batch Normalization for deep networks. Parent comment

Re:Batch Normalization for deep networks by gwern on Thursday February 22, 2018 @02:50PM (#16440791)
Initialization?

> W = np.random.normal(0, np.sqrt(2/(h.shape[0] + layer_dim[i])), size = (layer_dim[i], h.shape[0]))

A N(0, sqrt(2/width)) would produce negative values.

[ Parent | on HN ]
Re:Batch Normalization for deep networks by chillee on Thursday February 22, 2018 @04:00PM (#16441343)
I was talking about the graphs here: https://i.imgur.com/M6P71aC.jpg

I missed that he's not storing activations for those graphs, he's storing activations+batch norm. See my edit.

[ Parent | on HN ]