This is completely fair and rational. You shouldn't have to make any irrational sacrifices to work with people in far away places.
The way I look at this is - I am not (yet, perceived as,) good enough for people to want to work with me in spite of the distance/time difference etc. Lars Bak (for example) [1] is so good that he can work from somewhere in Europe and the rest of the team (and Google) make whatever compromises are necessary. In my opinion the best way to overcome this is to become (and be seen to be) way better than the average. "Be so good they can't ignore you" is good advice.
So, back to work!
[1] From Wikipedia
"In 2004, Bak was hired by Google to work on what was to become the Chrome browser. He did not return to the United States, preferring to work in Denmark where his daughters were also receiving their education. With a team of 12 engineers, Bak has been coordinating the development of the JavaScript machine for Chrome which he has code-named V8, to mirror the powerful automobile engine"
Doing this would be more work and would require actually having to employ people in these countries, not to mention consolidated accounts from three subsidiaries, with local accountants, audits etc.
I suppose it all depends on how the team is used to work and communicate, but I do wish we could have everyone in the same office more often. We have a team meeting in the same location once every six months for 4-5 days, but you can only achieve so much in a few days.
With everyone either having partners, children or both to take into account it isn't always so easy to fit everything together.