It seems like in this case you want to version API and program separately. API changes aren't the only thing people care about or that should be communicated in a version number. So semver would not just be "not perfect" but be actively misleading.