Thats not exactly true. Every 2016 blocks the network compares the actual blocks mined with the target blocks (at 10 mins per block), and adjusts the difficulty accordingly.

If you suddenly had less miners the current difficulty would mean that the time to mine a block would be greater than 10 mins. Depending on where it is in the 2016 block cycle it could also mean an extended period of time before the difficulty is rectified.

If the network suddenly dropped to a VERY low hashrate (like a single PC), it is possible it would take (at current difficulty) tens of millions of years to mine a single block - let alone 2016 of them. Of course, thats probably not going to happen, im just using it as an example to show that its not a fixed period of time per block.