If you start talking about true “infinity” then sure. You need to store enough state (or if you like, receive some input) for your computed output to be unique. And counting to infinity takes infinite amounts of memory. Infinity is not accessible to humans, and is best thought of as a thought experiment rather than a part of empirical reality.
But if you do 44kHz samples, a counter stored in a 32-bit integer that ticks up once per sample is enough state to give you all unique [i.e. non-looping] samples for about a day. A 64-bit integer will give you all unique samples for 13 million years. With a 75-bit integer you can get all unique samples for longer than the history of the universe so far.
So in practice, the “increasing” amount of memory required doesn’t (necessarily) grow fast enough to worry about.
I saw this trick and used it to animate different parts of an animation (each part every p frames, where p is a prime) and it works very well.
In theory one can construct a chaotic (this is a precise definition) function with very little data that could replicate that class of noise with finite data.
Infinite lists are also another beautiful construct in lazily evaluated languages.