IIRC that is one of Zeilberger's opinions. For myself, I don't think "biggest number" makes sense per se in mathematics, but it is a fact that no matter what resources we muster we're left with some numbers equaling things other than their successor when incremented.

My sense is that infinitary axioms like the existence of N, successor, induction, etc., basically allow mathematicians to write shorter and more straightforward proofs. They are likely to be sound until the cows come home, and moreover they represent what mathematics is essentially all about - finding some shortcut through the calculations we need to make, understanding what part of some structure or procedure is irrelevant.