Making a neural network say “I Don’t Know”: Bayesian NNs using Pyro and PyTorch

Re:Making a neural network say “I Don’t Know”: Bayesian NNs ... by paraschopra on Wednesday November 28, 2018 @12:41PM (#18553621)

My main point is not about aggregate accuracy, it's about comparing confidence across classes of datasets.

Edit: the entire point of bayesian approach is that you can make decisions on a loss function where you can make a tradeoff on the (business) cost of making a wrong decision and (business) regret of not making a decision. For real world applications, accuracy is a useless metric. What matters is the net (business) benefit or loss.

[ Parent | on HN ]

Re:Making a neural network say “I Don’t Know”: Bayesian NNs ... by chrisfosterelli on Wednesday November 28, 2018 @01:39PM (#18554287)

Your article bolds the following in the introductory paragraph:

> My final classifier had accuracy of ~97% on MNIST and it refused to classify white noise and the majority of unrelated (non-MNIST) images.

I'm saying that you are making a misleading claim when you say this. It is categorically untrue that you achieve 97% on MNIST with this approach. Accuracy on a standardized test set makes no sense when you select a subset from the test set for maximal accuracy. I'm not saying that your approach doesn't have value or make sense in some business cases, because I think the overall idea is interesting but these statements are not correct.

[ Parent | on HN ]

Re:Making a neural network say “I Don’t Know”: Bayesian NNs ... by pacala on Wednesday November 28, 2018 @05:37PM (#18556341)

Correct. That's why people use https://en.wikipedia.org/wiki/F1_score instead of just accuracy.

[ Parent | on HN ]