And really I suppose you don't ever need to go beyond version 1.x.y. Window Maker is quite a good example of this: latest version is 0.95.8 after 20 years of development.
If a library owner changes their API, all SemVer does is make sure you know about it. Are you saying that if they didn't follow SemVer they wouldn't change their API?
If a piece of software hasn't broken compatibility but still wants to tell people they've passed a big feature milestone, it's perfectly reasonable (and I think explicitly encouraged) to have separate user-facing and semantic versions - so you tell people it's "Foo 2.0" but publish it to npm as [email protected]. This is right and proper because the semver number exists solely to describe whether an update adds features and breaks compatibility or not; anyone using it otherwise is misusing semver.
I see far too much refactoring of JS library APIs that is essentially aesthetic across major versions - D3 and Angular spring immediately to mind (I understand the scalability justification for Angular; I just don't really believe it).
To me, having an elegant API is not the highest concern because what I'm interested in is building products, and from that point of view stability is much more important. I do not appreciate a committee of J Random JavaScript Developers randomly dumping extra work into my product backlog because they didn't like the cut of an API's jib, especially when these people have gone absolutely out of their way to drive adoption in the first place.
To give a counter-example: .NET code I wrote in the mid-noughties whilst working at Red Gate still runs today, substantially unmodified. Some of that stuff is 11 or 12 years old. SQL Dependency Tracker is the most obvious example, because the product hasn't changed much since 2006, except for additions to support new object types in new versions of SQL Server. The fact that new .NET versions haven't broken the product has obviously made it much easier to support through the years.