News for hackers. Stuff that matters.
Re:Does GPT-2 Know Your Phone Number? by colanderman on Sunday December 27, 2020 @10:34AM (#25551680)
This is something that has repeatedly bothered me while playing with GPT-2. It retains too much "long-tail" knowledge that seems counterproductive for a generic language tool. I would think those one-off associations should not be present after training has "digested" them but there they are.
Re:Does GPT-2 Know Your Phone Number? by aabhay on Sunday December 27, 2020 @11:29AM (#25552081)
Hard to decide how much is too much.. the hard (and interesting) part of language is in the long tail
Re:Does GPT-2 Know Your Phone Number? by alisonkisk on Sunday December 27, 2020 @08:46PM (#25555962)
It's over fitting, not long tail, that is the problem. It ends up copying input instead of analyzing and synthesizing.