My wife has made the same observation in her own workplace and refers to these people as 'the wanderers', although judging by their impact on productive output they should probably be identified as intruders.