It means a model which has a somewhat opaque internal working. A lot of modern ML approaches treat the model as a black box in that it is not particularly clear how the features actually interact to reach the prediction.

> “not particularly clear how the features actually interact to reach the prediction.“

This is quite true of many models, such as linear regression in the wild. You may also have a clear but wrong picture of how features interact, eg looking at coefficients of interaction terms in a misspecified linear model.

See for example, “The Mythos of Model Interpretability”

- https://arxiv.org/abs/1606.03490

Without a coherent definition of what “not black box” or “explainable” means scientifically, the buzzword of “black box” is also meaningless and is more of a political game to be a gatekeeper over what models are allowed to be used than any kind of honest intellectual inquiry.

I’ve worked professionally on systems where misspecified linear models and text models using simple ngram boosting were vastly more inscrutable than comparable neural networks or dimensionality reduction models for the same applications.

Nobody has any scientifically cogent idea of what makes a model “explainable” other than arguing semantics.

Quote:

"1. There’s a tiny functional language based on a small number of side-effect free combinators

2. For a given task, a program template (which the authors call a sketch), further constrains the set of programs that can be learned for the problem in hand. This also very handily constrains the search space of course, helping to make learning a suitable policy program tractable.

3. To help guide the search within the set of programs conforming to the sketch, a standard reinforcement learning algorithm is used to learn a (black box) policy.

4. The black box policy is used as an oracle (the Neural Policy Oracle), and a neurally directed program search (NDPS) tries to find the sketch-conforming program that behaves as closely to the oracle as possible."

The paper described:

http://proceedings.mlr.press/v80/verma18a/verma18a.pdf

...this seems kind of similar to the recent success in solving mathematical equations using deep learning. The language translation paradigm seems to have a lot more potential than some realized.

Merely decomposing a prediction into say a linear combination of predictors does not, itself, provide any type of explanation unless a variety of more complicated assumptions about statistical model checking turn out to be true.

Even worse, because the naive approach is to just treat those regression coefficients as if they do automatically give explanatory power or feature-wise attribution, people misrepresent things unwittingly and don’t carry out enough robust model checking, leading to wrong explanations that appear to have deceptive degrees of confidence associated with them.

I'm not sure we are talking about the same thing, or which of the three sources in my comment you are reacting to.

My understanding of "explainability" in this context is that it means "humans can understand why the model gives a certain prediction".

It seems like you are using it to mean "humans can understand why the thing modeled does something (presumably by referring to the model)". That isn't what I thought people meant by the term, nor does it seem like a reasonable goal to me.

Do we agree these are distinct ideas? Feel free to elaborate on where I've gone wrong.

Bullshit, any client you show a regression to can easily fully understand how the inputs change the output.

Section 4.1 of the paper he linked discusses extensively the limitations of linear model interpretability. Here's an example:

"With respect to algorithmic transparency, [the claim that linear models are more interpretable] seems uncontroversial, but given high dimensional or heavily engineered features, linear models lose simulatability or decomposability, respectively."

Given how many interesting problems are high-dimensional or solved through heavy feature engineering, you must be lucky to live in the happy space where problems are either low-dimensional enough to not need heavy feature engineering or where the client is already familiar with the high-dimensional features you're going to limit the model to using.

- http://www.saramitchell.org/achen04.pdf

The functional mechanism of a regression has nearly nothing to do with whether the regression explains the phenomenon it’s being used to model or not.