News for hackers. Stuff that matters.

A comment on Making a neural network say “I Don’t Know”: Bayesian NNs using Pyro and PyTorch. Parent comment

Re:Making a neural network say “I Don’t Know”: Bayesian NNs ... by chrisfosterelli on Wednesday November 28, 2018 @01:39PM (#18554287)
Your article bolds the following in the introductory paragraph:

> My final classifier had accuracy of ~97% on MNIST and it refused to classify white noise and the majority of unrelated (non-MNIST) images.

I'm saying that you are making a misleading claim when you say this. It is categorically untrue that you achieve 97% on MNIST with this approach. Accuracy on a standardized test set makes no sense when you select a subset from the test set for maximal accuracy. I'm not saying that your approach doesn't have value or make sense in some business cases, because I think the overall idea is interesting but these statements are not correct.

[ Parent | on HN ]
Re:Making a neural network say “I Don’t Know”: Bayesian NNs ... by pacala on Wednesday November 28, 2018 @05:37PM (#18556341)
Correct. That's why people use https://en.wikipedia.org/wiki/F1_score instead of just accuracy.
[ Parent | on HN ]