The semver docs explicitly address this scenario. If a change in minor update breaks compatibility release a new minor update to revert the change.

Semver is just _indicative_ attempt at describing changes.

If this wasn't true, we wouldn't have lockfiles.

Ie. semver is just approximate attempt at change description that aids/helps development/maintenance but should never be fully trusted.

The only way it could fully work in automated fashion is if the whole program would be written in some formal proof language – then dependency upgrades could be considered as breaking or non breaking. But again, easier and more precise from end user position, not author's position because breaking change in one project can always be non breaking in other if that part is not used/used in more relaxed manner.