Here (Link to the paper:[2]) this is different: The authors do NOT have access to the source code, they only have stripped binaries. From these unreadable binary instructions, they are able to identify interesting patterns: they extract the algorithm from the assembly. For this to work, they seem to need run time information. This means, as in conventional PGO, they need to have a wide variety of valuable input. It apparently cannot be done at compile time: "Current state-of-the-art techniques are not capable of extracting the simple algorithms from these highly optimized program."
Once the algorithm is figured out, the Helium framework is generating domain specific code in "Halide". Halide compilers knows how to optimize these stencil computation better than old hand written code, which give them these impressive improvements.
[1]https://msdn.microsoft.com/en-us/library/e7k32f4k.aspx
[2] http://groups.csail.mit.edu/commit/papers/2015/mendis-pldi15...
Dynamic instrumentation is powerful, I think Google is using DynamoRIO heavily. Optimising compilers for DSLs is a trendy topic because
1. We are beginning to understand well how canonically to embed DSLs in general purpose languages, and have to build compilers for languages.
2. DSLs typically work on much more restricted domains than general purpose languages, so optimisation is much easier, and more powerful.
The Pochoir compiler for stencil computations is an example of this. There are many open questions about compilation of DSLs other than optimisation, for example automatic generation of operational and axiomatic semantics of DSLs, and associated tooling. Really interesting research field.
[1] https://en.wikipedia.org/wiki/Profile-guided_optimization
[2] http://www.cs.virginia.edu/kim/courses/cs771/papers/bala00dy...
[3] http://www.dynamorio.org/