If you are outside of a tax jurisdiction, then yes, you shouldn't pay taxes inside that jurisdiction, because those are just taxes that wouldn't have been paid otherwise.

Are you paying 5+ digits tax and in the process of moving ? Then yes you can get the same special treatment while paying fraction of what you used to pay.