>I suppose it's remotely possible that I contracted HSV since then without ever having any symptoms

It actually is. Most people are asymptomatic, or their symptoms are so mild that they confuse them for a pimple.

https://www.cdc.gov/std/herpes/stdfact-herpes-detailed.htm