What's weird is when someone only looks at one spot. If you stare at people's eyes some will become uncomfortable, because naturally we make slight side movements. I'm sure the same it's with mouth. But most people can't tell if you're looking at their mouth or at them.

People do notice direct eye contact though. Maybe someone could explain this. I'm kinda curious why directly in the eye we notice but there's more wiggle room around the face.