Implementing gradient explanations for a HuggingFace text classification model

Re:Implementing gradient explanations for a HuggingFace text... by p1esk on Monday May 30, 2022 @01:18AM (#31555702)

How would you characterize the degree of explainability of large language models (e.g. gpt3)? Anything surprising or non-intuitive there?

[ Parent | on HN ]

Re:Implementing gradient explanations for a HuggingFace text... by behnamoh on Monday May 30, 2022 @02:16AM (#31555937)

i think a more general question is how would you measure explainability when you see it? is there some sort of metric for that?

[ Parent | on HN ]

Re:Implementing gradient explanations for a HuggingFace text... by visarga on Monday May 30, 2022 @02:41AM (#31556071)

One weak point I see - this tool only measures how much an individual input token would be changed to decrease the loss. The reason a token might have large gradients might be related to how many times it appears in the training set and how consistent is the training set with the evaluation set, not just how much the prediction disagrees with the target label.

So it jointly measures data coverage, consistency and data to target fit. Just my intuition, might be wrong.

[ Parent | on HN ]