It's not just an issue of parameter validation or well-defined calls; it's the interactions between calls where this is hard. It's issues of ordering, timing, when you can secretly skip synchronization because you know the API will synchronize just enough for you to get away without something more heavyweight, things like that. And then when you have paying customers? Yeah, you can't break their code, even if it's bad, because if you do you won't have customers for very long.
I think this is very naive/utopian.
Yes, be super focused in your design.
Yes, only expose APIs you have clear use cases for.
Yes, keep the surface area as small as possible.
Yes, have very focused requirements.
Yes, document the hell out of things.
Yes, implement strong parameter validation and other things to try to reduce the chance people do bad things.
Do all these things and more (these are all part of what makes exposing APIs hard work, that many people don't do).
But do not, for one second, believe that someone still won't do it wrong or abuse your perfectly designed API eventually... especially if it is successful.
Microsoft failed at keeping the surface area as small as possible. When their surface area expanded to a hack to deal with Adobe's hack of replacing code behind Microsoft's back, they went into very dangerous territory.
The flip side if Microsoft had bitten that painful bullet up front, then I believe that going forward companies like Adobe would have learned to not try to hack the system like they had. That would have given everyone, including Microsoft, a better reputation.
But of course Microsoft did not do that. And now Microsoft has such a history of nasty bugs that, even if Microsoft is in the right, nobody is going to believe it. (Which of course locks Microsoft into its current strategy of bending over backwards to accommodate everyone else's bugs, causing more bugs...)
Even if noone will ever use all three of the broken apps.
In this sense, maintaining backwards compatibility is simply a matter of reputation management.
Not a chance. People will blame Microsoft, because WhateverIUse worked perfectly before. The only parameter which changed is that they updated the OS, therefore the OS is to blame.
Behind almost every successful API is a big ugly mess of backwards compatibility to keep customers happy.
Don't do that. That way lays insanity. Be very, very clear up front that you will break backwards compatibility for those folks. Don't sweep it under the table, be very vocal about having done it. There will be short term pain as important customers (eg Adobe) learn the hard way that you really mean it. And long term relief as you don't have that legacy headache growing so quickly.
One estimate is that a 25% increase in requirements results in a 100% increase in software complexity. (See Facts and Fallacies of Software Engineering for a source.) That means that the complexity grows as the number of requirements cubed. Therefore the key to a good API is that it be short and simple. When you start to add implicit requirements on top of explicit ones, it quickly becomes long and complex, and the necessary software to support it and make future modifications becomes much worse.
This does not mean that designing APIs is not hard. But don't let your API become what is published and quirks that are not. Just don't.