Edit: the entire point of bayesian approach is that you can make decisions on a loss function where you can make a tradeoff on the (business) cost of making a wrong decision and (business) regret of not making a decision. For real world applications, accuracy is a useless metric. What matters is the net (business) benefit or loss.

> My final classifier had accuracy of ~97% on MNIST and it refused to classify white noise and the majority of unrelated (non-MNIST) images.

I'm saying that you are making a misleading claim when you say this. It is categorically untrue that you achieve 97% on MNIST with this approach. Accuracy on a standardized test set makes no sense when you select a subset from the test set for maximal accuracy. I'm not saying that your approach doesn't have value or make sense in some business cases, because I think the overall idea is interesting but these statements are not correct.