Well if you have to scale bigger and bigger for the payload, the cost of personnel that you need to have with a bigger craft starts featuring very negatively. Also if you can't operate from regular runways etc.

At some point on a degrading performance curve, you don't reach orbit at all even with zero payload.