Think of it another way: All matter in the universe is getting smaller. But the universe have the same size, so distances between everything increase.