Yes, but whoever hits the button last gets control, so you can take priority back.

(Unless you manage to hit the priority button for 40 seconds - but I assume that's for locking out a broken stick that's doing inputs at rest)