It's fascinating how embedding informational complexity in higher order relationships somehow gives rise to a map of represented concepts that are extendable with new data. Explainable NN models are going to be an absolute necessity - such as in the justice system or if we employ them in autonomous driving.

Tangentially, I don't believe GPT-N will be the way to achieve AGI, however we can learn much on said road travelled.

> or if we employ them in autonomous driving.

This is not looking to be the case right now.. Companies are doing quite well without explainable models. SDC companies like Waymo are hoping it will be sufficient to:

1. Produce a low failure rate estimate, based on millions of virtual miles/fake cities.

2. Combine several NN outputs in a logical way. That way there are at least some interpretable values that lead to the result. e.g. a separate model that produces candidate trajectories of surrounding vehicles, one to perceive sensor input, one to plan, etc.