You just need sub-millimeter precision, on a slippery glass remote made for babies with small hands, and then to time it so you touch the trackpad then click it such that you activate the skip back/skip forward function, but not so long that a cursor appears on the timeline. It's really fun, kind of like if your bank required you to do a little dance every time you wanted to buy something.

It’s probably your physical button. My button will audibly click when I have my thumb on the far left or right edge, but won’t actually trigger unless I press very firmly after hearing the click. Pressing the button with my thumb in the middle works reliably as expected. Apple support wouldn’t do anything about this except offer to sell me a replacement remote.

But I see the timeline controller on the screen react to it with the icon, it just doesn’t actually move in time.