Making a neural network say “I Don’t Know”: Bayesian NNs using Pyro and PyTorch

Re:Making a neural network say “I Don’t Know”: Bayesian NNs ... by jeremysalwen on Wednesday November 28, 2018 @11:15AM (#18552675)

I think that post is quite misleading. 90 and 100 are extreme values representing extreme confidence. Your model could just as easily learn [0.09, 0.10, 0.05] on the final layer, and the probabilities would then be [0.337, 0.340, 0.323]. And look, now softmax has the opposite problem! The final activation layers differed by a factor of 2, but softmax output almost identical probabilities!

Except this should not be surprising, because softmax is not scale invariant. Values on the unit scale is what you would normally expect, so if you feed it values 100x larger, it's going to be way more confident. If you feed it values 100x smaller, it's going to be way less confident.

[ Parent | on HN ]

Re:Making a neural network say “I Don’t Know”: Bayesian NNs ... by paraschopra on Wednesday November 28, 2018 @11:40AM (#18552968)

This is interesting. In theory, what you are suggesting can be true but in practice you do think the last layer can output 0.1, 0.09, etc especially when last layer is typically not pushed through sigmoid.

[ Parent | on HN ]

Re:Making a neural network say “I Don’t Know”: Bayesian NNs ... by rjtavares on Wednesday November 28, 2018 @12:27PM (#18553441)

Isn't that mostly true with highly curated datasets with clearly defined categories?

[ Parent | on HN ]