You absolutely should have gone on a date. Think of it this way, if by some chance it worked out you'd probably have the best "how we met" story of your entire social circle.
I feel that having a good “how we met” story is a big boost for a potential relationship. At least early on, when you have nothing else to go on. It says, “the odds of this meeting were low; this isn't a thing that happens every day, therefore it's more valuable”.
So it fizzled cause we weren't looking for, but looking "past" each other.
Moments of extreme awkwardness after flirting madly mutually verifying locational/timing/happenstance details followed by absolutely wrong physical descriptions of each other. Yah that conversation dried up fast.
I suppose it could have been a romantic start to some kinda relationship, but that ship sunk before it even had a chance.