It depends on your intent. If you are trying to be a creep and looking for "innocuous" ways to start moving in on an intended victim, then, no. If you are a decent human being with genuine respect for other people who isn't up to anything creepy, it's totes fine.
In fact, you should not be treating members of the opposite sex as "off limits" socially merely for being the opposite sex. This is a form of sexism and it insidiously closes a lot of doors.
So just be really honest with yourself as to why you are smiling at them or messaging them and if the answer is "I have plans that amount to sexual harassment," yeah, no. If the answer is "They are a coworker and that's how I interact with same sex coworkers," go right ahead. You're fine.
But the point at which you realize that you are a. Attracted and b. Trying to maneuver the person -- This is the point at which you need to have a heart to heart with your conscience. If you realize you want them and you don't really care if the feeling is mutual, then you are a creep. (The generic "you", not you the person I am speaking to at this moment.)
People do sometimes meet on the job, date, fall in love and get married. That can happen if both parties are equally aware of the potential pitfalls and both parties are equally looking out for the welfare of both people.
What separates a creep from an ordinary person who might happen to stumble into romance on the job is that creeps a. Make conscious and intentional plans aimed at a particular outcome and b. They do so with callous disregard for the other party.
Decent human beings who realize they have slippery sloped themselves into an awkward situation that might harm another start working to figure out how to avoid said harm. They don't have to be omniscient geniuses who can predict it ahead of time. They just have to decide to make other choices once the problem space becomes apparent.
> Is it appropriate to smile at a coworker of the opposite sex? Or to instant message them about work late at night?
No. And no-no.
That's sufficiently bizarre that I assume you misread the original post... (I'm applying Occam's razor).
Likewise, if the on-call engineer is female, I'm going to text or even call her late at night if I need her help.
Thing is, when I randomly message someone at night, I don't expect a response, nor do I view my recipient unfavorably if they don't respond until morning or at all. Of course, it's different if the entire team is working late, but that's only happened a few times in the past 2 years I've been at my current company.
They're not really assholes, but they get confused on how to act and may overcompensate. Is it appropriate to smile at a coworker of the opposite sex? Or to instant message them about work late at night? This can result in a lot of strange reactions.