You'd be surprised. Here are two real examples.
1) Project X 1.5.2 inadvertently returns `undefined` in some edge cases when it is documented to return (and previously returned) `null`. Due to the way people use the API, it's rarely encountered (hey they're both falsy) and nobody even bothers to report the discrepancy between docs and behavior until 1.7.2 is the current version. Unfortunately, some large projects that are very common use the actual behavior of 1.6.0 and are checking for a return `=== undefined`. Are you justified in breaking them?
2) Project X 1.8.0 does a big refactoring that eliminates tons of bugs and improves performance, and even passes extensive unit tests with flying colors. Unfortunately there are lots of "Ten things I learned by reading the source for Project X" blogs that describe non-public API behavior. And again, the projects using these behaviors are popular projects that many other people depend on. How do you deal with these breaking changes of undocumented functionality?
Software using Semantic Versioning MUST declare a public API. This API could be declared in the code itself or exist strictly in documentation. However it is done, it should be precise and comprehensive.
These can both be considered fixes. Undocumented functionality is not part of a public API.
1. It's documented to return `null`. The fact that it returns undefined in some edge cases is a bug. If you want to be nice about it, you can approach them and explain the upcoming behavior change as a heads up, however it's still a bug that should be fixed.
2. No sympathy for those projects here; you can't use undocumented/private interfaces and expect them to be officially supported. I do it myself, however I do it with full knowledge that every single version change is potentially breaking to me, and I make sure to have unit tests to confirm the behavior still exists in X version. I may also approach the project and say something like "Hey, I found using X undocumented interface is actually really useful, how can we expose this as an official thing?"
This is especially true in projects that don't have the resources to maintain a well done documentation site with the source.
It is also completely counter to attitude of the most successful open source maintainer, by many measures.[1]
These questions are tricky when you're building a popular OS with a lot of third-party software whose users will blame you if they break on a new release, even if the third-party software maker is actually to blame. They're a lot less tricky in a case like this, where you're providing a library for other programmers to consume.
And realize, that for many folks, they don't consider the version number of the product so much as they just consider the name.
It seems like, rather than attempting to document some of the complexity of API changes/versioning at an admittedly coarse grain, your argument is that we should stick with version "truthiness." I don't think I can get behind that.