Yep, just set boundaries. If your boundaries become a problem for someone higher up, they will raise it with you. Don't ask, just do.

Too many people assume they need to take meetings whenever and they'll put up with it quietly for years, never feeling confident enough to just assert themselves or set working hours in Google Calendar.