The problem is keeping the adversaries balanced. Otherwise you have one side overpowering everything (and typically not through the "correctness" of their argument, but scale and resources) and the adversarial system becomes detrimental to progress.