I think you are paying for the store front, the personal shopping experience, the models showing off the goods on runways far away. Advertisement and branding.

There are plenty of costs involved in luxury goods, not all of them goes towards material.