When I've used Story Points as the team manager, one of the things I made sure of was to completely avoid looking at any given team member's velocity, ever. I only cared about the team velocity, and only for the purpose of estimating risk to the timeline.

At the round-table people would report their retired points, and I'd add them to a running tally I kept on a piece of paper next to me. "2+1+0+5+2+2+8+0+1." Then I'd record the velocity for the week for the whole team, and we'd move on.