You can mostly avoid running into the single return people, if you sneak into and out of the building by the rear entrance.