Yeah, if I want to join it's easy to say "sorry can't make it there, but I'd like to join, can we reschedule?"

Sure that's annoying to the organizer, but everyone understands that finding a time where everyone is available is annoying and hard. If it's really hard to reschedule, you'll find some solution. "Sorry I can't, how about xyz attends and you get together after?"

It's really not that hard.