I suppose because sexual gratification is one of our species basest instincts, and due to the way our brain evolved to be social creatures, we have mirror neurons[1] which fire when we observe actions as well as when we perform them. For this reason, viewing sex can emulate (to an extent) our primal desire to mate.

Everyone has different levels of libedo however, so it's likely that your lack of understanding is due to your own lower libedo not generating the same level of sexual desire. Or, you are the kind of person where the emotional connection involved in sexual relations is far more important, which obviously means watching people you have no connection to would not arouse you as much. Alternatively, it could be that you may have decent access to actual sex, which is obviously more gratifying as a general rule.

[1]: https://en.m.wikipedia.org/wiki/Mirror_neuron