The only difference between the SBCL decls + noopt version and the decls + optimizations version, at least when I compiled it with SBCL 1.0.40.0, is the number of generated nops used to align the beginning of the inner loop which really shouldn't be noticeable at all. I'm guessing the 60ms difference between the two is just noise rather than the appearance of a difference in code generation.