Sure you can draw a good circle and say the ratio is Pi and that number encodes infinite information (albeit at high entropy) but to me that Pi is an algorithm for computing better and better approximations to a perfect circle, an object absent from nature.
I don't know QM but I suspect it is the same. It is our mental model. Using infinities is our concept. To me infinite things are algorithms (i.e. as X tends to infinity... Tends being an important word)
In any case, you can do lots and lots of mathematics with either only finite objects, or if you allow limits as you suggest, you can do almost all of math.
Only a vanishingly small part of math deals with actual infinities in a way that cannot be re-written in terms of limits.
However for natural numbers, the 'decompression' is very fast. Much faster than for Pi.
[0] You can make this precise via Bijective Numeration https://en.wikipedia.org/wiki/Bijective_numeration to handle leading zeroes nicely.
Quantum mechanics makes very good predictions.