The difficulty goes down as a direct result of less mining being done. Long term, blocks will be generated at the same rate, but each block would have less work behind it.