News for hackers. Stuff that matters.
Re:Does GPT-2 Know Your Phone Number? by 6gvONxR4sf7o on Sunday December 27, 2020 @02:30PM (#25553559)
Anything trained by gradient descent is in some sense a weighted combination lookup table (see domingos’s recent paper on everything being kernel machines, I’m on my phone so won’t try to find it). The crucial bit is finding an appropriate weighting. If you look up the nearest training data point and just use it to predict, you assume a notion of “nearest.” Getting that right is the trick.

So even if it all is mathematically equivalent to approximate lookup and approximate memorization, that doesn’t mean it’s “just” that.

Re:Does GPT-2 Know Your Phone Number? by antipaul on Sunday December 27, 2020 @02:41PM (#25553651)
Thanks for the reference. Here it is:

“We show, however, that deep networks learned by the standard gradient descent algorithm are in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel)“

https://arxiv.org/abs/2012.00152