News for hackers. Stuff that matters.
Re:Does GPT-2 Know Your Phone Number? by golergka on Sunday December 27, 2020 @10:20AM (#25551593)
I only took a beginner course in ML over 5 years ago, so this probably is a stupid question, but does this mean, the trained GPT-2 model encodes the source text into it's parameters somehow? Is this resilient — will it still remember it if we randomize the weights just a little tiny bit? Will it remember it if we clear a small portion of parameters?

Does the human memory work the same way?

Re:Does GPT-2 Know Your Phone Number? by jakear on Sunday December 27, 2020 @10:42AM (#25551725)
If you can recall text verbatim, the text has been encoded into your neurons’ connections. Tautologically.
Re:Does GPT-2 Know Your Phone Number? by minimaxir on Sunday December 27, 2020 @11:54AM (#25552288)
GPT-2 was trained with a Dropout of 0.1, which gives it a bit more resilience to noise and unusual input versus no Dropout.