You can do Gaussian filter in 2 passes: first along X, then along Y. Then you only grow linearly in size, rather than quadratically. This change should also improve your cache hits.