You can use thinner cables or have the same cables with less loss. As for actual component size it’s not that simple the higher the voltage the more clearance you need between conductors. So that is bigger airgaps in PCBs and higher rated insulation where gaps aren’t possible.

True, however, this can also be achieved by different insulating materials. In general, this is not really much of a problem; I agree, it needs to be considered.