> But there aren't a finite number of factories.

There absolutely is a finite number of factories.

There's probably even a finite, though perhaps quite large, limit on the number of factories that can exist (at least, within Earth's future light cone.)