At my current company we're actively discouraged in training from making small talk during interviews for exactly this reason. Even asking how their weekend was can lead to something like "Oh you played a few rounds of golf? I play golf too!" and you'll end up subconsciously judging them differently.