Social engineering is substantially about appealing to someone who can do all the steps you cannot perform from outside.

From what I understand of the literature, it’s often several interactions to gather enough information from several employees to learn to sound like you belong there, then using it all against someone with “keys” who escorts you the rest of the way.