Human-equivalent minds can obviously be implemented on top a framework that does not have inherently infinite or infinitely divisible values, because human minds are implemented on a top of a substrate that uses a finite amount of discrete neurotransmitters to do everything, and manages to work just fine.
If human minds can reason about infinite concepts using "hardware" where all the neural signals used to represent that concept are finite, then why should it be impossible a reinforcement learning system learn to reason about infinite concepts while using only finite numbers to form that representation?
This is like saying computers can't represent infinity because they have only finitely many bytes.
Suppose the treasury rewarded you a "superdollar", which is a special object that allows you to create any number of dollars that you want, on demand, as many times as you want. How many dollars would you say this superdollar is worth? Obviously, no finite number of dollars would be worth that one superdollar. The human mind can certainly understand the relative value of a superdollar vs. any number of dollars. That the human mind is implemented through finite electrochemical processes is irrelevant.
In a similar manner, the total amount of reward that you might ever get is capped by the amount of pleasure your brain can perceive at any given moment (which is finite) times your lifespan (which is finite).
You might reason and hypothesise about agents perceiving infinite rewards (as we are doing now), but this has nothing to do with the reality of homo sapiens rewards system(s), or, in fact, the rewards system of any agent existing in our physical reality, which is effectively bounded both in time and space.
Maybe you're right, but I don't think it's as obvious as you imply.
Consider this question. Can there exist hypothetical rewards x1,x2,x3,..., each one of which is significantly better than the previous, and at the same time another hypothetical reward y such that y is significantly better than each x_i?
If such rewards can exist, and if "significantly better" implies "at least +1 better", then that necessarily means y must be infinite.
Now, how would you argue that such rewards can't exist? If you say, "because then y would have to be infinite", then you're arguing in circles, assuming what you want to prove in the first place. It might well be that such rewards indeed can't exist, but if so, the argument would have to be different, and I suspect nontrivial.