You're technically correct, in the way that infinite growth is possible if we only limit ourselves to efficience gains, i.e. getting more value out of the same set of contrained resources. However, in practice, many things that people want cannot be obtained this way. Sure, a nicer chair or improved iPhone are good, but people also want summer houses and exotic vacations and a car for every family member and these things are massively resource-dependant.