News for hackers. Stuff that matters.
Re:We Found an Neuron in GPT-2 by xmcqdpt2 on Friday February 17, 2023 @07:15AM (#34833740)
They do and in fact it's relatively straightforward to show empirically on eg MNIST. The problem is that you need a much much larger network in the FCN case and thus need way more data and way more data augmentation to get a good result that isn't overfit to hell.

In the case of CNN the reason it works is that an image of an object X is still an image of object X if the X is shifted left or right. The property is translationally invariant. CNN are basically the simplest way to encode translational invariance.

Re:We Found an Neuron in GPT-2 by candiodari on Friday February 17, 2023 @07:57AM (#34834088)
> CNN are basically the simplest way to encode translational invariance

That's the geometric deep learning theory, isn't it? Do you know if there's a list somewhere of exactly what invariance has which ways to simulate it? Like an overview?