Even if someone's driving defensively, sudden changes in the environment lead to a more dangerous situation. If they're driving defensively, hopefully they can deal with it, but that is no reason to create dangerous situations in the first place.
However, in this constructed situation, it seems to me that the driver is being more unsafe by driving that fast.
However, in places with vehicles going differing speeds, I would put the onus of safety on the faster vehicles.
I have been paying attention in a city for a while. I have seen the situation you describe, but it's rare in my experience, even with SF supposedly full of reckless fixie-riders. Dangerous driving is far more common in my experience.
And dangerous driving is probably more common (certainly by number of instances), and it should be condemned as well.
And then look at the resulting injuries.