If one really wants to do it the slow and hard way, it's possible to compute a random permutation of n elements one element at a time with n bits of storage (less with fairly generic compression techniques): just remember the set of numbers that have been produced so far and sequentially pick one of the remaining ones. You'll waste some combination of additional memory (for arithmetic decoding state) and random bits (for rejected samples), but materializing a large randomly permuted vector should be slower.

Good point. Storing a number of magnitude n! will take n log n bits, and storing n numbers up to n will also take n log n bit.

In other words, good old Fisher Yates, storing the resulting permutation, and a simple lookup is probably the way to go.