Batch Normalization for deep networks

Re:Batch Normalization for deep networks by singularity2001 on Thursday February 22, 2018 @03:30PM (#16441093)

also there are 'leaking' ReLUs

f(x) = a if x<0 else b

usually 0 < a << b

[ Parent | on HN ]

Re:Batch Normalization for deep networks by rvarma on Thursday February 22, 2018 @05:31PM (#16442142)

I actually think the idea of using leaky ReLUs is interesting, because it'll still provide a small gradient when x < 0, which perhaps may slightly alleviate the vanishing gradients issue

[ Parent | on HN ]

Re:Batch Normalization for deep networks by chillee on Thursday February 22, 2018 @03:59PM (#16441335)

I'm aware. He's using ReLU in this case though.

[ Parent | on HN ]