It might help if you think of it as directing the camera instead of moving the image. When you click and drag left, you're telling the camera to look to the left.