News for hackers. Stuff that matters.
Re:A New Approach to Understanding How Machines Think by goldenkey on Monday January 28, 2019 @08:13AM (#19017147)
Just finished reading the paper. TLDR; a linear classifier is trained on the activation values of an already trained network when it is fed examples of a specific high level feature. Assuming the feature is isolated to a certain linear subset of the activation matrix, a vector along the decision boundary can be computed. When feeding examples, the magnitude of the directional derivative in this vectors direction can be computed on the activation matrix, yielding a measure of how much the feature is responsible for the overall derivative..ie. the total activation.

Seems like an obvious technique. A few months back, someone posted a similar linear reverse engineering technique to make a customizable face generator from already trained GANs.

Re:A New Approach to Understanding How Machines Think by jeromebaek on Monday January 28, 2019 @08:32AM (#19017271)
"how much the feature is responsible for the overall derivative" part is obvious. But I'm curious how the assumption can be justified - how do you isolate a feature to a certain linear subset? And how do you know that this feature is (i.e.) "stripes"? The mapping from vectors to human language is the part that seems hard.
Re:A New Approach to Understanding How Machines Think by goldenkey on Monday January 28, 2019 @09:30AM (#19017619)
By training a binary linear classifier over the vector space of the other NNs responses to a set of inputs with that specific feature.

Language shouldn't be necessary if your feature can be conveyed through examples.

But yes, it's a big assumption to say that all features can be isolated as linearly decidable subsets of the activation space.

I would guess one could get better results with stronger, non linear classifiers combined with more abstract generalizations to directional derivatives.