They seemed to have addressed most of the fairness and "What about language X on platform Y using framework Z" kinds of questions with "Send us a pull request and we'll include it in the next benchmark".
I'm tempted to tackle this even though I don't currently have the time.
[1] http://www.techempower.com/benchmarks/ [2] https://github.com/TechEmpower/FrameworkBenchmarks
I believe they have plans to add additional tests which will allow for caching and that kind of thing.
I don't think that removing caching across all frameworks equally is creating 'artificially hamstrung implementations' as all the frameworks would benefit from caching.
I'm not saying it would be easy (or economically viable), but it's certainly possible. Somebody should try it.
This isn't an automateable task, although it may be a crowdsource-able one. Luckily the benchmark game publishes pretty much all of the tooling they use, so you could probably get a pretty big headstart there on your improvements.