I'd recommend setting up some Shortcuts with the addresses that can be triggered from Siri. For example, "Siri, we're going to the office", "Siri, let's go swimming",

One nit with this is that I haven't been able to make it automatically transition into navigation.