Another way to think of it:

Convnets are a method to speed up learning by sharing filter-coefficients. In other words, one could just as well train the network by replacing the convnets by fully-connected layers, except that of course training would take a lot longer.

If one views convnets as purely a computational optimization, then the approach of the paper can be considered a "hack" that only works because of this optimization.