Good plastic is not flimsy.

Polycarbonate, an almost commodity plastic, is really strong and light. The next step: composites are way better than any metal if you want a good strength-to-weight ratio. The best, most expensive bikes are made out of carbon fiber composite for a reason.

Plastic is commonly used as casing material for stuff designed for rough environments. Portable power tools, PA speakers, even guns. So if something designed to be used on the battlefield can be made of plastic, there is no problem for headphones worn, at worst, in the subway.

The only problem with plastic is that it suffers from being cheap, as in, not expensive enough. People want expensive products to be made of expensive materials. Also, subconsciously we associate weight with quality, so much that cheap manufacturers sometimes add metal inserts to their otherwise low quality products to trick people into thinking they are better than they really are.

That it to say, unless you put a premium on look and feel (nothing wrong with that), don't hesitate to buy plastic headphones, they can last you a lifetime as long as the right plastic is used at the right place.