The goal of changing the difficulty is to maintain a constant rate. As the computational power dedicated to mining Bitcoins increases, the difficulty increases proportionately, to keep the overall rate of Bitcoin discovery more or less constant.

If tomorrow, everyone agreed with this guy and quit mining, the next difficulty adjustment would make it easier to mine Bitcoins, to keep the constant rate with a smaller pool.