If you are driving during rush hour where everyone is stressful, then idiots will flash headlights, do risky behaviour, etc, that's true. And yeah, driving slightly over the speed limit to be consistent with the traffic around you is generally a good idea. But if they are too much over the limit, it's their doing, not mine, and if the risky behaviour causes an accident in which I'm involved, it won't be my fault if I'm following the rules. Conversely, no judge will accept it as an argument if you speed because of the idiots and you cause an accident.
I personally just let the idiots be idiots. Especially when the limit has a cause, e.g. a construction site or something: they put a lower speed limit for the safety of the construction workers. I won't compromise their safety, not a bit.
But on (sub)urban streets driving the speed limit decreases the chance of causing serious injury to other road users, including pedestrians, even if it may lead to an increase in the total number of crashes between motorists.
I feel like it's the opposite... When I drive the speed limit, other drivers around me become more aggressive. Even if they have to exit the highway 20 seconds later they fiercely fight for tiny speed differences of 10km/h, start flashing their headlights, etc.