HSV-1 and -2 transmit through contact from skin location to skin. Including through asymptomatic shedding. Only sometimes is that location covered by a condom - in location and in timing.

See also: Wrestlers' "Herpes gladiatorum" which is just HSV-1. Which demonstrates how HSV-1 survives for transmission far more easily or broadly than just genital contact. Is there "HSV-2 gladiatorum"? Apparently it's usually HSV-1 but probably yes - neither HSV-1 or HSV-2 are really location specific.