I can’t look into your use case, but I’m not sure you realize how spectacularly bad xoring with a fixed value is as a way to generate a ‘random’ permutation.

For example, it will still alternate odd and even numbers.

Also, an adversary can derive the xor key from a single sample, and predict the entire sequence from it.