News for hackers. Stuff that matters.
Re:Retrospective review of Gödel, Escher, Bach (1996) [pdf] by taeric on Monday March 29, 2021 @02:14PM (#26625964)
The article making the rounds a few weeks ago about rethinking general thinking feels relevant. By showing that many large neural networks are ultimately really good at memorizing the training set, in curious how much the two views you are showing are in conflict.

It is the age old "rote memorization" versus "learning". In large, I suspect those do not have a clear line between them. Such that emergent behaviors are expected and can be taught.

Re:Retrospective review of Gödel, Escher, Bach (1996) [pdf] by anaerobicover on Monday March 29, 2021 @02:37PM (#26626288)
Could you link the article you mention? I missed it making its rounds, but I would love to read it.
Re:Retrospective review of Gödel, Escher, Bach (1996) [pdf] by taeric on Monday March 29, 2021 @03:00PM (#26626669)
Certainly, https://news.ycombinator.com/item?id=26346226 is the post I was thinking of. Pretty sure the article is older.

Do let me know if I misrepresented it. I thought it was a clever way to show that the models are not finding inherent relations in the data, by just randomly labeling all training data and having the same speed/accuracy on the random labels as on the real ones.

Edit: I worded the above poorly. They showed the models were capable of merely learning the training data. Despite that, they also showed that moving the trained models from test to validation did not have a linear relationship with the amount of noise they added to the data. That is, the training methods seem indistinguishable from memorization, but also seem to learn some generalisations.

Re:Retrospective review of Gödel, Escher, Bach (1996) [pdf] by anaerobicover on Monday March 29, 2021 @03:37PM (#26627139)
Thanks! I will read it later.