Have you tried balancing this urge to return, against self-anger and humiliation at all the time and gas you're wasting by driving back to check?

When I get irrational urges, I make the urges battle my insecurities and get out of the way. Insecurities usually win.