Fuck that yeah, do what you want, fuck what people think about who you are or what you're doing. You'll at least give them a good story to tell about that nut who walked up to that guy on the subway and said, "I dig that jacket, wanna get some coffee together?"
Who cares what these people (who are judging you for meeting someone in public) think?
They're very unlikely to be anyone meaningful if they're doing that, since being a good conversationalist is a trait that most successful people possess.