No, it changes in proportion to the number of blocks generated in the past week or two. This could even result in a decrease if the number of miners decrease.
The difficulty is adjusted every 2016 blocks based on the time it took to find the previous 2016 blocks. At the desired rate of one block each 10 minutes, 2016 blocks would take exactly two weeks to find. If the previous 2016 blocks took more than two weeks to find, the difficulty is reduced. If they took less than two weeks, the difficulty is increased. The change in difficulty is in proportion to the amount of time over or under two weeks the previous 2016 blocks took to find.
If tomorrow, everyone agreed with this guy and quit mining, the next difficulty adjustment would make it easier to mine Bitcoins, to keep the constant rate with a smaller pool.
However the pool didn't remain the same in the past. All the BTC press (and $10 USD rate) substantially increased the number of people mining. This sped up the discovery of bitcoins but it also gave each miner a smaller cut.
At this point the only way to mine a non-trivial amount of BTC is to invest $ in a GPU-heavy mining rig.