As the system increases in complexity, you need more and more measurement/monitoring in order to gain intuition for how the system works. For example, watching Ganglia graphs of mapreduce clusters make it 10x easier to tune system parameters.