Or the majority of the people that live somewhere will have all "air rights" controlled by the city, and dictate how tall things can be built. "Air rights" aren't a thing unless the jurisdiction makes it a thing; you can't charge a toll for planes passing overhead.
Why would you "bargain" with someone who doesn't own anything that you want? You're talking about compromise, not bargaining, and that involves a friendly relationship; the kind normal people living in a home don't have with anonymous developers.
You can up to at least 365 feet, and sometimes higher depending on jurisdiction.
I am sure that the vast majority of garden owners would be willing to accept a couple thousand dollars in exchange for having slightly less sun hitting their house.
If the apartments threaten to block the sun for your garden, is it really totally irrational to care?
You might think gardens are stupid, but you've got to admit gardens generally require sunlight!