There used to be an old-fashioned, well-known hand-signal for this situation. You pause long enough to hold up one finger for a second or two, then return to what you were doing. The polite would then poke around or sit down or return in a half-minute or minute to see if you'd paused. The impolite would get barked at.