It seems easy enough to agree some kind of core - "everyone comes in 2-4pm on Wednesday and we do any meetings then" or some such. I mean, all-remote teams manage to get work done.