IMO, if you're going fast enough and there isn't room for cars to pass, you should be riding down the center of the lane. Especially if there's hazards in the bike lane.

Don't let people do stuff that compromises your safety.