But yes, tools like this would eliminate most of the semver issues.

I can't believe such logic isn't built-in to more package managers.

  def foo():
    return True

  def foo():
    return halts("some turing machine program")

Of course, one can still likely get useful answers of "definitely incompatible" etc., with much more tractable analyses. AIUI, the Elm version ends up just looking at the function signature, and catches things like removing or adding arguments: for appropriately restricted languages, it is likely to even be possible to determine if downstream code will still compile, but that's not a guarantee it will behave correctly.

  Version 1:
  add Int -> Int -> Int
  add x y = x + y

  Version 2:
  add Int -> Int -> Int
  add x y = x * y

Perhaps with some sort of functional extensionality you'd be able to compute compatibility perfectly, but I can't imagine that ever being feasible in practice.

That being said, what Elm does offer is still a huge improvement over humans trying to guess whether they made any breaking changes :)

But behavior testing is undecidable, easily follows from halting theorem. I doubt it would even be recognizable.

Even with a test, you may not find this. In the IOException example, the author calls out why:

> When we upgraded, all our tests passed (because our test fixtures emulated the old behavior and our network was not unstable enough to trigger bad conditions)

The only way to catch this type of thing is to emulate the entire network side of things, and that's still only as good as your simulation of the real world. Again, reality is even if you test your upstream to this extent, you're probably mocking a bunch of things, and that may mask something in a way you won't see until possibly production use.

Nitpick: you are not using SemVer.

A normal version number MUST take the form X.Y.Z where X, Y, and Z are non-negative integers, and MUST NOT contain leading zeroes. X is the major version, Y is the minor version, and Z is the patch version. Each element MUST increase numerically. For instance: 1.9.0 -> 1.10.0 -> 1.11.0.[1]

[1]https://semver.org/#spec-item-2

Some of your version numbers lack the patch version.

Suppose you're writing a library in Python. Everything in it is public. Even the dunder class members are, because it's just name mangling, and the language spec even documents what exactly it does!

Now, is anyone going to seriously claim that every single identifier in every Python library is part of its public API, and any change that affects it is a breaking change? Because that's certainly not the "expected social norm".

Granted, Python is a somewhat extreme example. But in practice, this also comes up in languages like Java and C#, when dependencies are more intricate than what the access control system in those languages can fully express.

And then there are backdoors:

> What can happen, tends to happen. The actual code actually existing always has the final say.

You can use Reflection to access any private field of any object in Java. There's actual existing code doing that in practice, too. Does it have the final say, and does it mean that internal representation of any Java class in any shipped Java library has to be immutable, so as to not break the API clients?

The API is 100% intended for internal use only, but someone insists on ignoring that and consuming the private API anyways. Instead of forcing them to write their own headers which silently break at runtime when function signatures change in certain calling conventions which don't check those signatures, I instead allow them to include headers with a few keywords like "private", "internal", "do_not_use", or "i_am_voiding_my_semver_warranty" in the path, perhaps only after they make some similarly scary #define s, so it's at least a build failure.

I see this a lot in Java libraries, for instance.

So I usually make most stuff public as such, but put internals in a namespace/scope that makes it clear that these are implementation details. Relying on implementation details always carries the risk of breaking when upgrading.

This allows for a lot of flexibility when needed, while also not polluting the "truly public" API.

It’s a bit like a house on a corner with a big front yard. People may cut through the grass to save time but you can’t blame the homeowner when he finally puts up a fence.

I've never found standards to be all that standard.

I think the right solution here is to build version numbers off of your black box tests.

The devil is in the details though. What’s a change to the tests mean exactly? Adding new tests is probably a patch release. Modifying tests demands at least a minor version number, but what makes it a major version number? Deleting tests probably qualifies. Removing assertions probably does too.

But now what if the author is bad at tests too? Can I substitute my own? (I could do that right now for regression testing, and in fact I do occasionally for interdepartmental issues).

Well, many do. You wouldn't believe the number of broken HTTP clients/servers out there.

The difference is that most HTTP clients/servers have to work with some significant subset of the pre-existing HTTP infrastructure (otherwise why would anyone use them?) and so that constrains their implementation to be "mostly correct". It's literally network effects :).

Libraries on the other hand usually start out with a single consumer and have no such constraints on their implementation so you end up with various levels of ossified brokenness or breaking non-ossification.

In any case you need to read through the changelog or (absent that) the actual code diff and think about how you use the application. It's not 100%, but no versioning system will be able to identify how you use code and how you expect the semantics of an application to be.

Ideally, there would be a tool that can inspect your entire codebase and determine if the change is "breaking". This still has issues if the change lies outside of your codebase (perhaps such as changing the configuration of your AWS services).

Or use ComVer

https://github.com/staltz/comver/blob/master/README.md

>When we upgraded, all our tests passed (because our test fixtures emulated the old behavior)

Doing that, upgrading your dependencies and expecting everything to work just because those tests passed? That's naivete.

If they'd built decent integration tests that used the actual library (instead of "assume nothing changes" fixtures) and made more of an effort to simulate realistic scenarios then their tests probably would have flagged up most of the issues they had.

Alas, this seems to be one of the side effects of following the "test pyramid" "best practice".