Why do CNNs generalize so poorly to small image transformations?

Re:Why do CNNs generalize so poorly to small image transform... by fallingfrog on Sunday June 10, 2018 @07:24PM (#17280692)

Time shift invariance, which is used when you assume that past patterns predict future results, is one form of invariance. Space shift invariance and rotation invariance are others. You have to specifically program your nn to look for time shift invariance; stands to reason you'd want to design your architecture for the spacial invariances too. In other words: it's not reasonable to expect a neural net to derive space and rotation invariance from first principles, and I'd be very surprised to learn that the human brain didn't have special purpose hardware for accomplishing those things.

[ Parent | on HN ]

Re:Why do CNNs generalize so poorly to small image transform... by magicalhippo on Monday June 11, 2018 @11:09AM (#17284886)

Having dabbled with image processing, your comment reminded me of the Fourier Mellin transform, which can be used for translation, rotation and scale invariant feature detection.

I did a quick search and came up with this paper[1], and while they don't use the Fourier-Mellin transform, they do use a log-polar transform[2], where rotation and scaling are transformed to translations. This, they claim, result in much improved classification of rotated and scaled data.

I haven't been keeping up on the latest in the ML field tho, so maybe it's rubbish, but like you I'd be surprised if some extra processing doesn't play a role.

[1]: https://arxiv.org/abs/1709.01889

[2]: https://sthoduka.github.io/imreg_fmt/docs/log-polar-transfor...

[ Parent | on HN ]

Re:Why do CNNs generalize so poorly to small image transform... by dr_zoidberg on Monday June 11, 2018 @09:47AM (#17284122)

You made me think of the Margaret Thatcher Illusion, for which the best example I found was Dr. Phil Plait[0]. Funny thing, the idea of Capsule Networks (Hinton et al)[1, 2] seems to tackle some of these issues, though it's they're a bit young yet and more more work.

[0] https://www.opticalspy.com/opticals/dr-phil-plait-optical-il...

[1] https://arxiv.org/abs/1710.09829

[2] https://hackernoon.com/what-is-a-capsnet-or-capsule-network-...

[ Parent | on HN ]

Re:Why do CNNs generalize so poorly to small image transform... by fallingfrog on Monday June 11, 2018 @10:00AM (#17284227)

I think that convolutional networks are really mostly a way to address space shift invariance too; and I even wonder if the same job could be done by some special purpose code that just runs the underlying neural net with a bunch of different rotations of the same image. That's probably how they do it now.. I feel like that's probably close to the optimal approach.

[ Parent | on HN ]

Re:Why do CNNs generalize so poorly to small image transform... by candiodari on Tuesday June 12, 2018 @11:00PM (#17299909)

> the human brain didn't have special purpose hardware for accomplishing those things.

How about a head mounted on a ball joint ? Would that count ? (since obviously it would introduce small spatial variations VERY often).

You can make a similar argument for being mounted on a mobile robot (like the human or any other animal body).

[ Parent | on HN ]