When it comes to the overall quality of the image, counting both the size of the sensor and the absolute diameter of the aperture is counting twice. The amount of light that hits the sensor, for a given field of view, is entirely determined by the absolute diameter of the aperture. Bigger sensors collect more light because you can use a bigger absolute aperture for a given f-stop and field of view. Or if you prefer, a larger absolute aperture collects more light for a given f-stop and field of view because you can use a bigger sensor.

Obviously, the format has implications for DoF, but those can be either positive or negative. When I shoot 4x5, I spend most of my time fiddling around with movements trying to get everything in focus :)