Re:Does GPT-2 Know Your Phone Number?
by colanderman on Sunday December 27, 2020 @10:34AM
(#25551680)
This is something that has repeatedly bothered me while playing with GPT-2. It retains too much "long-tail" knowledge that seems counterproductive for a generic language tool. I would think those one-off associations should not be present after training has "digested" them but there they are.
[ Parent
| on HN ]
Hard to decide how much is too much.. the hard (and interesting) part of language is in the long tail
Re:Does GPT-2 Know Your Phone Number?
by alisonkisk on Sunday December 27, 2020 @08:46PM
(#25555962)
It's over fitting, not long tail, that is the problem. It ends up copying input instead of analyzing and synthesizing.