I never got around to actually implementing the following in the PathScale compiler, but my idea was to add logging in all the places where allocations do something, and then create lists of optimizations that I expect to have successfully fired in various functions or programs. This would be especially handy for checking that new optimizations didn't disable important things for various SPECcpu benchmarks.

Sorry, "the places where optimizations do something". Didn't notice until after the edit window was long passed. I blame autocorrect :-)

Well, you could embed a machine-checked proof that instruction sequences of the targeted form will sometimes survive previous optimization passes and therefore the given optimization will always be contributing something.

But that's hard, so every compiler I've used leaves it to human engineering instead.

You can emit stats for time and # of instructions changed by each pass. gcc certainly does the first, don't remember the second.

You can do both kinds of tests: check that a pass still optimizes some GIMPLE or RTL input the same way, and check that the C code is optimized the same way. In turn, the latter can either look at per-pass dumps, or at the final assembly code for one or more architectures.