oh I should also add that the clustering (kmeans) is very fast. The analogous part to the inner loop is "feature_vectors". Which has 2 main cases: case 1, in the case nothing has been computed it calculates the corner response images at all the different spatial scales (big operations). case 2, feature_vector just selects data from the corner images for the pixels demanded.

Now my algorithm is sparse so its normally jsut selecting a subset of pixels, although as it calculated the spatial responses of the whole image it doesn't really make any odds to my runtime.