Re:Announcing Rust 1.7
by Terribledactyl on Thursday March 03, 2016 @04:59PM
(#11220387)
> ...effectively using Rust's public package repository as an extended test suite.
This is one of the coolest and most practical things I've read in a while, seeing what happens to stable(ish) real wild code. So many practical applications and analytics are coming to mind in many areas.
Thank you. Sparks my interest in Rust again.
Should also have a look at http://stackage.org (quite similar but in Haskell, yet curated -- a "try to compile all" feature also exists IIRC).
Not sure how much this is used in order to test impact language changes.
It regularly is.
I believe Perl and CPAN originally coined this approach. I'm glad to see from the parent and other comments that this approach seems to have caught on in other ecosystems as well.
Scala also has something similar, called the community build: https://github.com/scala/community-builds
It does not contain everything, but (open source) authors are encourage to add their library to the mix.
Coq has that too! :) https://coq-bench.github.io/clean/Linux-x86_64-4.02.3-1.2.2/...
I think that something similar is also done by the Chicken Scheme folks:
This has been one of the many benefits of quicklisp (a repository for common lisp packages); there is a test-grid maintained by Xach and he reports any breakages when there is a release candidate of a new sbcl.
Does Crater ever find unstable tests that don't pass consistently on a good baseline? If so, do you just blacklist those packages and/or feedback improvements?
NM: after reading more carefully, it's clear that at least in the case where rust causes the "regression" you feedback the improvements, so likely the same for unstable tests.
Well, the "tests" here aren't actually tests—they're whether the package compiles or not. If the compiler is being inconsistent, that's a pretty big bug ;)
Here's Crater's regression report for the linked soundness fix: https://gist.github.com/nikomatsakis/2f851e2accfa7ba2830d#ro... . It detected four root regressions, which means that there were four packages that were relying on unsound behavior. This isn't necessarily the full extent of the regressions, however, because if we ship a compiler that breaks those packages, then any other packages that have the formerly-unsound packages as dependencies will obviously also break.
Having a concrete list of regressions also allows us to go through the ecosystem and submit PRs ourselves to bring the affected packages back to building, which is usually quite easy. Crater is a really, really great tool.