Should also have a look at http://stackage.org (quite similar but in Haskell, yet curated -- a "try to compile all" feature also exists IIRC).
Not sure how much this is used in order to test impact language changes.
It regularly is.
I believe Perl and CPAN originally coined this approach. I'm glad to see from the parent and other comments that this approach seems to have caught on in other ecosystems as well.
Scala also has something similar, called the community build: https://github.com/scala/community-builds
It does not contain everything, but (open source) authors are encourage to add their library to the mix.
Coq has that too! :) https://coq-bench.github.io/clean/Linux-x86_64-4.02.3-1.2.2/...
I think that something similar is also done by the Chicken Scheme folks:
This has been one of the many benefits of quicklisp (a repository for common lisp packages); there is a test-grid maintained by Xach and he reports any breakages when there is a release candidate of a new sbcl.
This is one of the coolest and most practical things I've read in a while, seeing what happens to stable(ish) real wild code. So many practical applications and analytics are coming to mind in many areas.
Thank you. Sparks my interest in Rust again.