> This is why there is a decline button next to the accept button.

Depends on your MacOS version.

In Mojave there’s actually no meaningful way to interact with an invite through a notification. There’s a dropdown but it’s not reachable. It only appears if you mouse over where the “x” button should be, which causes the button and dropdown to appear. If you move the mouse off this button both the button and the dropdown vanish.

Also, I’m afraid to click that “x” because I’m still not sure if it dismisses the notification or declines the invite.

And I still can’t figure out how to decline from the calendar with a message. I have to go to my deleted items in Mail and reply to the invite there.