Those aren't necessarily all that massive, though. Enormous in size, yes, but the mass would be a lot less than the mass of the fuel for less efficient rockets like chemical or nuclear thermal.

Either way, any vessel travelling beyond Mars or Venus in a time short enough to be safe for humans is going to require in orbit assembly, so size in terms of volume becomes less of a constraint. But as long as we're boosting all of our mass from Earth, that's what costs the money.