I've decided 100% of papers saying their modification of a neural network is interpretable are exaggerating.

Personally, I'm looking forward to MNNs: Mansplainable Neural Networks.

I think the case for interpretability could have been made better, but in Figure 3 I think if you look at the middle "prototype" rows from the traditional vs Tversky layers, and scroll so you can't see the rows above, I think you could pick out mostly which Tversky prototype corresponds to each digit, but not which traditional/linear prototype corresponds to each digit.

So I do think that's more interpretable in two ways:

1. You can look at specific representations in the model and "see" what they "mean"

2. This means you can give a high-level interpretation to a particular inference run: "X_i is a 7 because it's like this prototype that looks like a 7, and it has some features that only turn up in 7s"

I do think complex models doing complex tasks will sometimes have extremely complex "explanations" which may not really communicate anything to a human, and so do not function as an explanation.

It's wishful thinking.

Neutral networks need to be over parameterized to find good solutions, meaning there is a surface of solutions. The optimization procedure tries to walk towards that surface as quickly as possible, and tend to find a low-energy point on the surface of solutions. In particular, a low energy solution isn't sparse, and therefore isn't interpretable.

Interesting; can you provide some references for this way of thinking?