Take it a step farther and the photon isn't even truly nessicary. The recoil experience when the photon was emitted, and the push felt when it is absorbed should match exactly, and essentially correspond to a single, equal, collision event to the right observer.