To elaborate, we use a tool called Crater to estimate the potential impact of breaking changes. It attempts to compile all of the packages on crates.io with a new version of the compiler to determine any regressions, effectively using Rust's public package repository as an extended test suite.

Here's Crater's regression report for the linked soundness fix: https://gist.github.com/nikomatsakis/2f851e2accfa7ba2830d#ro... . It detected four root regressions, which means that there were four packages that were relying on unsound behavior. This isn't necessarily the full extent of the regressions, however, because if we ship a compiler that breaks those packages, then any other packages that have the formerly-unsound packages as dependencies will obviously also break.

Having a concrete list of regressions also allows us to go through the ecosystem and submit PRs ourselves to bring the affected packages back to building, which is usually quite easy. Crater is a really, really great tool.

> ...effectively using Rust's public package repository as an extended test suite.

This is one of the coolest and most practical things I've read in a while, seeing what happens to stable(ish) real wild code. So many practical applications and analytics are coming to mind in many areas.

Thank you. Sparks my interest in Rust again.

Should also have a look at http://stackage.org (quite similar but in Haskell, yet curated -- a "try to compile all" feature also exists IIRC).

Not sure how much this is used in order to test impact language changes.

It regularly is.

I believe Perl and CPAN originally coined this approach. I'm glad to see from the parent and other comments that this approach seems to have caught on in other ecosystems as well.

Scala also has something similar, called the community build: https://github.com/scala/community-builds It does not contain everything, but (open source) authors are encourage to add their library to the mix.

Coq has that too! :) https://coq-bench.github.io/clean/Linux-x86_64-4.02.3-1.2.2/...

I think that something similar is also done by the Chicken Scheme folks:

http://wiki.call-cc.org/eggref/4/salmonella#introduction

http://tests.call-cc.org/

This has been one of the many benefits of quicklisp (a repository for common lisp packages); there is a test-grid maintained by Xach and he reports any breakages when there is a release candidate of a new sbcl.

Does Crater ever find unstable tests that don't pass consistently on a good baseline? If so, do you just blacklist those packages and/or feedback improvements?

NM: after reading more carefully, it's clear that at least in the case where rust causes the "regression" you feedback the improvements, so likely the same for unstable tests.

Well, the "tests" here aren't actually tests—they're whether the package compiles or not. If the compiler is being inconsistent, that's a pretty big bug ;)

Ah, I thought it was doing "cargo test" to see if the compiled code had a functional regression. But that would be challenging to investigate regressions (especially in the face of unstable tests).

For those that don't follow Rust very closely, I sent in numerous (well, something like "30") fixes to various libraries after this landed, and the vast majority of them involved adding ": Sized" in the correct place. So while it was true that it was breaking, it was easy to update.

We take breaking changes ridiculously seriously, and soundness is basically the only case where they make sense to do.

Why does Rust not use semver to clearly communicate when breaking changes do take place?

Disclaimer: I'm not a Rust author, but I actively follow the development process.

Rust does use semver. But as the concept of "breaking change" is ill-defined – as I've understood – in the definition, the Rust project is using the following heuristic:

* If it's a safety/soundness-releated bugfix, it's considered a minor (1.x.0-changing) change even if it makes some existing crates stop compiling. Because the crate was relying on broken/unsound functionality, that's acceptable.

* Impact of all such breaking changes are measured and assessed against the ecosystem using the Crater tool.

* Even if crates stop building, the changes should be such that it's trivial to fix them. (In the best case, one-liner type annotations etc.)

Btw. Rust sometimes deprecates APIs that have been supplanted by better alternatives. Deprecations are warnings, and they are allowed to go away only on a x.0.0 version change. Additionally, careful consideration for stability and breaking changes is done when designing new language features.

Rust does use semver and takes it very seriously. RFC 1105 for example documents exactly what is considered a breaking change to stabilized library APIs. The Rust team has an explicit allowance for breaking changes related to type soundness or memory changes without performing a semver bump. This is the first such change, there is one more coming in the future.

Both changes are very small; they do not impact most users and the users who are impacted can easily fix them unless they are exploiting the existing "bug" to do something unsound.

https://github.com/rust-lang/rfcs/blob/8e2d3a3341da533f846f6...

So, this is kind of a complex topic. The sort of TL;DR is, Semver itself does not actually specify things, it just says "compatibility". Specifically,

  > MAJOR version when you make incompatible API changes,

But does not define what "compatible" means.

In a statically typed language, virtually any change, including an addition, can be construed as a "breaking change". For any proposed change to Rust, I could write code that that change would break. Adding a new method is a breaking change, for example, because I could have written a method with that name previously.

Note that this is also consistent with other languages: Java, for example, regularly makes very small breaking changes, even though it's widely regarded as a language which takes compatibility extremely seriously.

So, with that definition of 'breaking' not being particularly useful, we've done what you're supposed to do with Semver: lay out what 'compatible' means. The primary documents for this are https://github.com/rust-lang/rfcs/blob/master/text/1122-lang... and https://github.com/rust-lang/rfcs/blob/master/text/1105-api-...

I quote:

  > This RFC proposes that minor releases may only contain breaking changes
  > that fix compiler bugs or other type-system issues.

Without a formal language spec, and with certain bits of the language being generally under-specified, you can language lawyer forever about a term like "breaking". That's why we went through all of the possible ways that we could change the language, and spelled out what kinds of versions we're allowed to make those changes in. I would argue that Rust is far more clear about its version guarantees than many, many other languages.

It's important to remember that semver is a _social_ tool to communicate roughly what has changed. If this blog post were about a Rust 2.0, you would rightfully assume that it's going to be tough to upgrade to. That's not the case here. The actual impact of this particular change to the ecosystem is virtually nothing.

> Adding a new method is a breaking change, for example, because I could have written a method with that name previously.

Then you should have used a way to namespace your method, as is done in Objective-C with category methods.

All methods are namespaced but there is sugar which can cause a conflict because it searches the namespaces in scope for methods.

Imagine the trait (namespace) `Baz` defines a method `bar()`, and `Foo` is a type which implements `Baz`. `foo.bar()` is sugar for `<Foo as Baz>::bar(foo)`. Imagine `Foo` also implements `Quux`, another trait in scope, and `Quux` is extended to also define `bar()`. Now `foo.bar()` is ambiguous, because it could also mean `<Foo as Quux>::bar(foo)`. As you can imagine, though, this happens very rarely.

You can't avoid a conflict unless you require specifying the namespace at every call-site, which is in my estimation a much greater burden than editing your code to use the explicit namespace form if a conflict arises because of a new method in a dependency.

We have a module system, but that doesn't help. You can still define methods on arbitrary types, subject to the coherence rules.

You can still write out a disambiguated form to distinguish the two, but it's not the default way. That's one of the reasons why adding a defaulted method to a trait is a minor change. See https://github.com/rust-lang/rfcs/blob/master/text/1105-api-... for more.

> Then you should have used a way to namespace your method, as is done in Objective-C with category methods.

Categories do nothing to protect against breakage due to adding methods.

It does.

Remember, any bugfix is a breaking change if someone was relying on the buggy behavior.