Re:The Pile: An 800GB Dataset of Diverse Text for Language M...
by lambdaphagy on Saturday January 02, 2021 @07:56PM
(#25618296)
Bias in prediction, rather than parameter estimation, is a perfectly well established sense of the term. In particular, people doing language modeling are practically never concerned with identifiability, because you can't pick out one weight out of a trillion parameter model and say what it ought to be in the limit of infinite data.
Re:The Pile: An 800GB Dataset of Diverse Text for Language M...
by 6gvONxR4sf7o on Saturday January 02, 2021 @11:08PM
(#25619343)
But when people use the term bias in NLP, that’s what they’re talking about. They don’t want an aspect of the model to do something it ought not do. It’s a case of omitted variable bias causing things like the word analogy issues you hear about. Not an issue of bias in predicting the masked word.
It’s not opinions I disagree with, it’s aspects and behavior I don’t want, which is the statistical sense.