If what you say is true, that's an argument in favor of eg aggregating via median instead of average.