They already are?

lol. Having done the laundry, I think women's clothing has significantly higher cost:weight ratio.

That's comparing apples to oranges.

Mostly they don't.

if you are very tall they charge a lot extra for having tall sizes... what are you talking about??

In most cases the cost of fabric itself is a pretty minor part of the garment price- you're paying for someone to design the clothes, assemble them, ship them, and operate a store that sells them, and those costs are pretty much the same for small and large sizes. Adjusting the price based on the amount of fabric used would probably end up being a dollar or so for the things most people wear on a daily basis.

Unusually large or small sizes can end up more expensive (and/or only manufactured in limited quantities) because they're not commonly bought and they take up space on the shop floor and in inventory which could be used for things with higher turnover. (Edit: Also at the extreme ends of sizing simply enlarging or shrinking the pattern won't work well, you have to redraft it so it sits correctly on a petite or plus-sized frame).

Shipping and even assembly costs should scale with size.

Some shoes (Meindl Boots) actually go up in price for larger sizes (>46 EUR, 13 US I think) due to the additional cost of material.