Guards know this too, and given the choice between letting someone through who you are pretty sure is legitimate and potentially starting a fight which will generate a complaint, they'll "do the best they can" and just wave people through.
Motivations interact in funny ways, and the parameters aren't like software: they're hidden and surprising.
Look at government and large corporate entities for how to address this. Visitor conference rooms are on the ground floor or away from the office environment. There’s a receptionist who controls access to the employee area, and identification is captured and associated with a staffer for both signin and signout.
I found out later they’re given total immunity from offending someone so they have no fear of doing their job.
I was a bank teller during college, and would be occasionally berated by people when I'd ask to check their ID when they were withdrawing money. I always asked if they'd prefer I let anyone trying to take money out of their account do so w/o checking photo ID. One guy swore he was going to get me fired for not letting him take money out of his account because he forgot his wallet at his desk.