Re:Sometimes, requiring explanability is counterproductive
by naresh_xai on Thursday February 06, 2020 @12:02PM
(#22257781)
Not to mention the fact that most models are trained without a background class and tend to give overconfident predictions on out of distribution samples.
Finding something odd for an algorithm (especially a deep neural network) is hard because they fail in just so many ways. For example, lenet for mnist almost always gives high confidence predictions for random tensors(torch.randn). Most imagenet models fail in the presence of just 20-30% salt and pepper noise. (Both of these are problems solvable through simple preprocessing techniques)