My solution has been to use the Fastmail website to accept calendar invites, but use khal and vdirsyncer to manager the majority of my calendar. As far as contacts, I believe Fastmail can import and export to vCard, which can be managed locally with khard and vdirsyncer as well. I use khard directly with mutt to access email contacts and I've been happy with it.