The key to decidability or undecidability is whether diagonalization is possible, not whether or not there are disguised references to infinity somewhere.
On the other hand Peano arithmetic is not only about infinite objects, and very many more than just the naturals because it is rich enough to allow you to encode other ostensibly more sophisticated infinite objects in it, it is itself an infinite object. It can't be reduced to a finitary decision procedure the way weaker arithmetics can.
Diagonalization is accounted for by my second example of conceptualizing infinity: you can't do a diagonalization argument unless you contract a variable. In particular, you can admit full unrestricted set comprehension if you can't contract to derive absurdity. Referencing section 2.3 here [1]. It was this analysis of Russell's paradox that led to the discovery of light linear logics, or so the story goes.
Godels incompleteness theorems can be seen as a statement about the gap between finite and infinite mathematics. Decidability, semidecidability and undecidability can be seen as the relationship between boundedly quantified arithmetic statements, statements with one unbounded existential quantifier, and statements with one unbounded universal quantifier.
Another avenue of exploring the gap between finite and infinite mathematics is via linear logic. There the thesis is that contraction, the logical reuse of variables, is where infinity creeps into logical reasoning. Indeed logic without contraction is quite tame. Logic with unlimited contraction is wild. Surprisingly there are logics with an intermediate strength of contraction: so-called light linear logics. These can classify reasoning that embodies polynomial time computation or elementary time computation. So in another sense infinity can be measured by algorithmic complexity.