Whether something is breaking or not is not always clear-cut. For one thing, changes can become breaking in retrospect sometimes. For example, in C#, renaming a method argument wasn't breaking until the language introduced named arguments in calls.

There are also changes which are breaking in a very non-obvious way, to put it mildly. For example, in C# again, adding any new member to a class can break existing code that happens to pass implicitly typed lambdas to overloaded methods, because overload resolution involves checking whether the lambda body is valid for a given candidate - thus, adding a member can make a lambda ambiguous. I'm not aware of anyone actually treating this as a breaking change for semver purposes, though.