It's a mistake to assume the marginal cost scales linearly with payload. Fuel will (ish), but many other costs are fixed per launch.

I'd argue your estimate is not aggressive. SpaceX continues to claim an eventual marginal cost in the high 7 to low 8 figures per launch. I'm inclined to believe that deep analysis over a back-of-the-comment Fermi estimate.