At my last company (US on the east coast) I was the only remote member of the team, and the 5 hour time difference, although relatively small, and with significant work-day overlap, was still very difficult. Having to wait until the afternoon for a code review, staying online to support my team later.

It did mean that my mornings were totally free and I could get a lot done - so it swings both ways, but outsource everything to Europe isn't the panacea some people assume it to be.