Not necessarily the same thing, but have you looked into the Unison project at all?

Unison is an open source functional programming language based on a simple idea with big implications: code is content-addressed and immutable.

Homepage: https://www.unisonweb.org/

Strange Loop talk: https://www.youtube.com/watch?v=gCWtkvDQ2ZI

Check out Elm. The package manager already forces you to bump your semver when you make some kinds of changes. It's not perfect, but it gets you most of the way there and helps narrow down some of the discussion about what is a breaking change and what isn't.

> But if we have tried to define 'compatible' in a rigorous way, I haven't seen it

My understanding (and the way I've versioned my projects) is that a major version denotes a stable API and behavior you can safely rely on. What functions are exported, what arguments they take, etc. That's a pretty easy thing to not break.

Things that _aren't_ covered are implementation details such as the location of files in the package, the reuse of objects internally, or other not explicitly-defined behavior.

A couple of examples:

1. In JS, you can import from sub-folders: `let const = require('somePkg/a/b/c)`, but that depends on internal organization. That's not guaranteed across versions.

2. You can also add properties to an object. If we export a function that takes a request object and returns a response, we don't guarantee those are identical objects. We _do_ ensure that the object returned from the function has properties x,y,z, but not that anything you defined before calling the function will still be there.

We would have this back-and-forth on how to version fix releases. "Well, if someone was depending on this buggy behavior, this update will break their code". Ultimately, we said "this is what we guarantee will work. For anything else, run tests anytime you upgrade".

We're not perfect (all software has bugs), but this has worked pretty well.

The problem is that just about everything breaks the API. I have had situations where obscure bug fixes break my app because I was depending on the behaviour of that bug.

The answer IMO is to just do a full test of your stuff after doing updates. I update many packages at once to make the most of my time.

Then the problem is with you. You wrote code that had a dependency on a specific non-guaranteed feature (the bug) and then feel jilted b/c they changed the non-guaranteed feature. What you should have done was written defensive code and tests around the NGF so that when it changed, your tests would catch that and either not upgrade or allow you the chance to fix it.

I don’t feel any negative feelings towards the library developers. I have my versions locked so these unexpected surprises don’t happen. In this case it would have been extremely difficult for me to have noticed that the behaviour seen was not intended and supported by the developers.

My point is not that this is an unsolvable problem, but that it’s not a good idea to go “semver says this is non breaking, let’s just chuck it in production”

I once wrote a python module that applied diff patches as a part of the deployment process. This let us 'fix' libraries we depended on without having to maintain a full fork until the maintainer fixed the bug. If a new version came down, we could test the patch and update it if needed.

Necessity, meet invention...

People have toyed with this sort of thing to verify API compatibility in Rust, because you can quantify all the sorts of changes that are considered compatible and incompatible, e.g. adding a type is compatible, changing the return type of a method is incompatible. Though in some cases it’s surprisingly difficult to decide whether a change should be considered compatible or not, as in Rust it’s possible for adding a method to a trait to break downstream code by introducing resolution conflicts—so should you allow adding methods to a trait, or should you consider that backwards-incompatible? And if you have a prelude which will conventionally be used as `use some_crate::prelude::*;`, adding an item to that can cause conflicts, so is adding public items not backwards-compatible after all? (This is one of the arguments against glob imports.)

I hope that eventually it will be possible to confirm API compatibility (however it is defined) as part of making a release. Whether that can be done on the crates.io server or not is another matter, because it could vary by platform, building could require local dependencies, &c. This isn’t full tests, of course, just API, so there’d still be scope for error.

There's been a lot of attention given to this in Rust:

- Defining semver requirements for the ecosystem: https://stackoverflow.com/questions/41185023/what-exactly-is...

- There's also a library that attempts to automatically check sermver adherence of a crate: https://github.com/rust-lang/rust-semverver

- And there has been quite a bit of effort into preventing semver requirements from fracturing the ecosystem. This revolves around the compiler working with multiple major versions of a single library: https://github.com/dtolnay/semver-trick

I also like the idea of extending the "standard" tests with your own "custom" tests if your app happens to rely on some undocumented behavior (Hyrum's Law). Then you get a nice error message when that observable behavior (not part of the contract) breaks, instead of the app falling over.

Tests aren't the actual requirements doc once your code is used by a nontrivial number of people. People will always depend on your undocumented implementation details. Always.

Here’s the thing though. If the test suite declares the API, I could always write my own test suite that is more precise, which does cover the things I use “off label”. I may get caught a couple times, but eventually I’ll get it sorted.

Being able to do that depends on whether I can assign such things after the fact, or if the author has to do it.

The version of the API and the version of the artifact that implements it are distinct, not the least because the artifact can implement several (major) versions of the API.

Using tests across versions is a definitely a trick to consider but your coverage may vary. Probably better to do it in addition to other methods.

Yes, that’s basically what I was thinking. New test suite runs against old code, new tags are created if the tests pass.

As for code coverage, that is always a problem, and not one to sneeze at.

So follow me here. I do a lot of tech selection for teams. Not just which libraries have he features we need, which ones are going to result in. One pair programming due to human error.

If I see that the poorly documented (accidental?) feature I care about has poor code coverage, should I use it? Should I use it and file a PR with pinning tests?

I think in this case the complexity of the interface tests will need to be as complex as the code itself, so you are back to the same quality issues but with the test framework itself.

The Haskell's Stackage people has some interesting discussions about this on their lists. (Caring about types too, obviously.)

TLDR, it doesn't change much, what means it doesn't create a lot of problems, but doesn't really solve any either. Versioning is a humane concept, and you can't solve it with pure tech, you'll always need guidelines and procedures.