The way I see it is that we appreciate order, structure, logic, symmetry, yes? This is true for painting, drama, computing, nearly all art. I don't see why you make an exception for music. Maybe it's because it doesn't contain spoken language, or because it is abstract. I submit that any beauty you can derive from architecture, sculpture or industrial design, you can derive from music as well, for all of the reasons relating to the qualities I've outlined. The only difference, as you aptly put in your comparison to drugs and sex, is the "delivery mechanism" of the artistic content. Thus, I cannot see a reason to discriminate and single out listening to music as a waste of time.
I think that if more people appreciated the complexity and richness of classical music, chances are they would forgo other what I call cheap products in our world: bubblegum pop music, underdeveloped political arguments, and terribly unusable technology. Good music is on your side of intellectual achievement. Listening to and understanding good music is the mark of high artistic and intellectual standards, a good, positive willingness to explore and expand one's aesthetic horizons. I think the PRODUCT of this attitude, is far from, as you say, some kind of negative counterproductiveness, where time spent listening to music is time not spent doing science. The product of this kind of attitude is a high standard of achievement that is completely compatible with doing great science with passion and dedication. At least that's my view.
I assumed that making something of value for other people is what makes life meaningful. Making and shipping a product, supporting a family, writing poetry and producing music, doing research and participating in constructive discussions is meaningful for me. I assumed that some readers here share these values to some extent.
I didn't tell people what to do. I shared my view and am genuinely interested in your view on the subject. I tend to think that is why we are all here.