Well, from a technical perspective that is indeed the case. Once they physically have your contact info they may do as they please.
You, as the iOS or Android user, are not giving them permission to use your contacts "properly" or "nicely"--you're giving permission to access them, the raw data of all of them, and once that's done all bets are off. If the app is untrustworthy it is free to go crazy (one of the reasons I always say "no" to that question).
I don't see how Apple or Google can stop this in a technical way without making the permissions more fine grained which in turn makes it more confusing to users (who probably mostly click "OK" anyway).
Apple could, however, make better app policies so that they can pull apps when they attempt this kind of shady crap. I'm not familiar with the Android app store policy, so I won't speculate there.
By your logic, it would be completely useless to even read the fine print, because giving them access to the addressbook would imply my consent for them to do anything technically possible with it.