It’s probably your physical button. My button will audibly click when I have my thumb on the far left or right edge, but won’t actually trigger unless I press very firmly after hearing the click. Pressing the button with my thumb in the middle works reliably as expected. Apple support wouldn’t do anything about this except offer to sell me a replacement remote.